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Chapter  1 


INTRODUCTION 

The  selection  and  funding  of  an  R&D  portfolio  is  a  difficult 
and  important  decision  process  for  any  organization  which  is  involved 
in  research  and  development  tasks.  The  future  business  base  and 
the  relationship  of  the  organization  to  its  environment  are  determined 
by  the  type  and  level  of  effort  of  current  R&iD  projects.  If  an  organi¬ 
zation  does  not  adequately  fund  current  research  efforts  having  the 
highest  possible  benefits,  it  could  find  itself  lagging  its  competitors 
in  the  future,  losing  its  market  share  or  present  status,  and  possibly 
not  surviving  in  its  present  form.  For  a  private  company,  unemploy¬ 
ment  and  economic  depression  for  the  local  area  could  be  the  result, 
but  for  the  U.S.  military,  the  results  could  be  detrimental  to  the 
country's  survival. 

The  U.S.  Air  Force  (AF),  as  well  as  all  the  other  services, 
has  to  be  continually  assessing  the  perceived  threat  to  the  nation  and 
determining  how  to  meet  that  threat  in  the  most  effective  and  efficient 
manner  possible.  AF  managers  have  to  be  able  to  evaluate  all  cur¬ 
rent  and  proposed  R&D  projects  with  respect  to  the  current  AF  goals 
and  objectives  in  order  to  effectively  and  efficiently  allocate  resources 


to  those  projects  which  provide  the  greatest  benefit  to  the  security  of 
the  United  States.  Since  all  resources  are  limited,  the  AF  has  to 
compete  with  all  other  federal  agencies  for  funds.  The  ultimate 
challenge  for  AF  managers  is  to  provide  the  nation  with  the  weapon 
systems  which  yield  the  most  security  per  dollar  spent. 

This  study  is  concerned  with  an  investigation  into  the  pro¬ 
gram  evaluation  and  ranking  process  used  by  the  Defense  Wide 
Management  and  Support  (DWMS)  Mission  Area  Review  panel  at  the 
Aeronautical  Systems  Division  (ASD)  for  the  1982-1986  Program 
Objectives  Memorandum  (POM).  The  purpose  of  this  ranking  process 
is  to  develop  a  list  of  programs,  within  the  DWMS  mission  area, 
based  upon  their  contributions  to  AF  objectives.  This  list  represents 
the  panel  members'  preferred  order  for  funding  programs  within  the 
DWMS  mission  area.  Evaluating  DWMS  programs  with  respect  to 
AF  objectives  is  a  difficult  undertaking  because  DWMS  programs  are 
support  programs  and  contribute  to  the  objectives  of  several  or  all  of 
the  other  mission  areas.  The  overall  intent  of  this  investigation  is  to 
develop  recommendations  and  conclusions  which  can  aid  the  DWMS 
review  panel  during  the  next  POM  exercise. 

Aeronautical  Systems  Division 

The  Aeronautical  Systems  Division  is  one  of  the  four  product 
divisions  of  the  Air  Force  Systems  Command  and  is  tasked  with  the 


mission  of  planning  and  managing  the  acquisition  of  all  aeronautical 
weapon  systems,  subsystems,  and  equipment  for  the  United  States 
Air  Force  (ASD/AV,  1980:3).  In  performing  its  mission,  ASD  has  to 
supervise  many  types  of  programs  at  different  stages  of  development. 
The  majority  of  programs  under  ASD  control  are  either  advanced  ■ 
development,  engineering  development,  or  procurement  programs 
with  very  little  of  the  actual  detailed  design  work  on  a  system  per¬ 
formed  at  ASD.  The  engineers  and  managers  at  ASD  supervise  con¬ 
tractor’s  performance;  design  and  write  specifications  for  system 
design  and  operation;  supervise  cost  and  schedule  performance;  and 
manage  conflicts  between  costs,  schedule,  and  product  performance. 

All  of  ASD's  programs  can  be  grouped  into  the  following 
seven  mission  areas:  strategic  offense,  strategic  defense,  air-to- 
surface  attack,  counterair,  reconnaissance,  mobility,  and  defense 
wide  management  and  support  (Weber,  1980a).  AFSC  Regulation  80-2 
states  that  ASD  will,  for  each  mission  area: 

.  .  .  evaluate  each  R&D  program  assigned  to  the  Pro¬ 
duct  Division  and  provide  to  HQ  AFSC/XR  (Developmental 
Planning)  each  year,  before  the  AFSC  POM  submission,  an 
overall  integrated  Product  Division  R&D  program,  including 
relative  rank  of  each  program  by  mission  area  as  supported 
by  analysis  and  rationale  [AFSCR  80-2,  1978:2], 

This  Product  Division  R&D  program  evaluation  and  ranking  process 

is  the  lowest  level  of  zero-based  budgeting  within  the  Department  of 


Defense. 


Three  different  levels  of  management  at  ASD  are  involved  in 


the  decision  package  ranking  process.  To  initially  rank  the  decision 
packages.  Mission  Area  Review  panels  are  established  for  each  one 
of  the  seven  mission  areas.  These  panels  evaluate  and  rank  all 
decision  packages  within  their  assigned  mission  area.  Next,  the 
Program  Review  Group  reviews  and  makes  appropriate  changes  to 
the  proposed  rankings  within  each  mission  area.  Finally,  these  pro¬ 
posed  rankings  are  briefed  to  the  Decision  Review  Group  which  is 
chaired  by  the  ASD  Commander.  Upon  approval  by  the  Commander, 
the  rankings  are  forwarded  to  AF  Systems  Command  as  ASD's  input 
to  the  POM  process  (ASD/A\£,  1980:15). 

AF  Systems  Command  incorporates  the  mission  area  pro¬ 
gram  rankings  from  ASD  with  the  mission  area  program  rankings 
from  all  other  agencies  within  AFSC  and  determines  one  ranking  for 
each  mission  area  which  is  AFSC's  input  to  the  AF  POM.  The  Air 
Staff  has  to  incorporate  the  AFSC's  programs  with  all  other  Air  Force 
programs  in  order  to  develop  the  AF  POM. 


Defense  Wide  Management  and  Sur 


The  purpose  of  the  Defense  Wide  Management  and  Support 
mission  area  is  to  provide  for  support  programs  that  are  common  to 
one  or  more  of  the  combat  mission  areas  and  to  improve  the  quality 
of  that  support  while  trying  to  reduce  operations  and  maintenance 
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costs  (Weber,  1980a).  This  mission  area  is  composed  of  many  differ¬ 
ent  types  of  programs  which  support  the  combat  mission  areas  and 
provide  indirect  contributions  to  the  elimination  of  deficiencies  in  the 
AF.  Not  only  can  one  DWMS  program  contribute  to  the  accomplish¬ 
ment  of  more  than  one  mission  area's  objectives  but  also  that  program 
can  contribute  various  percentages  in  each  applicable  mission  area. 
The  process  of  evaluating  the  contributions  of  DWMS  programs  to  AF 
needs  and  making  trade-off  decisions  between  programs  in  order  to 
rank  them  for  the  POM  is  a  very  complex  decision  process. 

Statement  of  Problem 

The  DWMS  panel  members  interviewed  indicated  that  they 
need  a  more  efficient  method  developed  to  evaluate  and  rank  all  the 
programs  within  the  DWMS  mission  area  for  the  POM.  Several 
issues  which  are  inherent  to  the  problem  of  having  an  inefficient 
evaluation  and  ranking  process  are  included  in  the  following  questions 
which  will  be  addressed  in  this  research.  Does  the  information  about 
each  program  provided  to  the  panel  represent  the  most  current  and 
accurate  description  of  that  program?  What  documents  are  available 
to  aid  the  panel  members  in  evaluating  the  contribution  of  each  pro¬ 
gram  to  the  needs  of  the  AF?  What  is  the  best  method  of  judging  the 
benefits  from  each  program?  Knowing  the  benefits,  what  is  the  best 
method  of  allocating  budget  resources  to  the  optimal  portfolio  of 
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projects  in  order  to  maximize  the  benefits  to  the  Air  Force? 

Statement  of  Objectives 

The  objective  of  this  thesis  is  to  evaluate  the  process  that 
the  DWMS  panel  used  in  developing  the  198  2-1986  POM  and  to  develop 
improvements  to  the  process  in  the  following  four  areas: 

1)  collecting  information  from  the  program  managers  about 
their  programs; 

2)  using  AF  documentation  to  tie  DWMS  programs  to  the 
long-term  goals  of  the  AF; 

3)  evaluating  and  measuring  the  benefits -to  the  AF  from  the 
programs;  and 

4)  selecting  a  portfolio  of  projects  within  given  budgetary 
constraints . 


Methodology 

During  the  1982-1986  POM  exercise,  the  DWMS  Mission 
Area  Review  panel  had  to  evaluate  seventeen  programs  containing 
about  seventy  decision  packages  and  rank  the  decision  packages 
according  to  their  benefits  to  AF  objectives.  The  panel  members 
used  a  scoring  model,  which  contained  four  criteria,  to  evaluate 
each  individual  decision  package  and  then  obtained  a  total  decision 
package  score  by  summing  the  individual  scores.  Next,  the  decision 
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packages  were  ranked  according  to  their  total  scores.  Adjustments, 
based  upon  the  panel  members'  expert  opinions,  were  made  to  the 
initial  ranking  if  the  panel  felt  that  this  ranking  did  not  accurately 
reflect  the  true  relationship  between  decision  packages .  Finally,  the 
cumulative  cost  for  adding  decision  packages  according  to  the  ranking 
structure  was  determined  and  compared  to  the  budget  figure.  All 
packages  whose  cumulative  cost  was  less  than  or  equal  to  the  budget 
figure  were  included  in  the  recommended  project  portfolio  while 
projects  whose  cumulative  cost  was  greater  than  the  budget  figure 
were  excluded.  Chapter  4  contains  a  detailed  analysis  of  the  ranking 
process  used  by  the  DWMS  panel. 

The  purpose  of  this  research  is  to  evaluate  the  effectiveness 
of  the  process  the  DWMS  panel  used  during  the  POM  exercise  and  to 
recommend  improvements  for  the  1983-1987  POM  exercise.  Initial 
discussions  with  the  DWMS  panel  chairman  and  initial  reviews  of  the 
POM  guidance  documentation  indicated  that  four  major  areas  needed 
to  be  investigated.  The  first  area  concerns  the  concept  of  zero-base 
budgeting  in  the  federal  government  since  the  ranking  process  per¬ 
formed  by  each  Mission  Area  Review  panel  is  the  lowest  level  of 
ZBB  at  ASD.  The  second  area  of  investigation  is  concerned  with  the 
availability  of  AF  documentation  which  relates  AF  programs  to  AF 
objectives  by  mission  area.  The  panel  relied  only  on  the  information 
contained  in  the  decision  packages  and  the  program  managers' 
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briefings.  The  next  two  areas  deal  not  with  the  collection  of  project 
information  but  how  to  use  that  information  to  evaluate  and  select  an 
R&D  project  portfolio.  Once  the  most  relevant  information  is  pro¬ 
vided  to  the  panel,  what  is  the  best  method  to  combine  that  information 
with  the  expert  opinions  of  the  panel  members  in  order  to  realistically 
determine  the  expected  benefits  of  a  decision  package  to  the  AF? 
Finally,  given  the  expected  benefits  for  each  decision  package,  what  is 
the  best  method  (scoring  models,  economic  models,  mathematical 
programming  models,  or  other  approaches)  to  select  an  R&D  project 
portfolio  which  maximizes  the  total  possible  expected  benefits  to  the 
AF? 

In  order  to  evaluate  the  decision  package  ranking  process 
used  by  the  DWMS  panel  and  how  it  relates  to  ZBB  the  following  cri¬ 
terion  was  used:  how  well  did  the  DWMS  panel  apply  the  theory  of 
ZBB  to  the  R&D  project  portfolio  problem?  In  order  to  evaluate  the 
criterion  three  sources  of  information  were  used.  First,  a  literature 
search  was  conducted  into  the  theory  of  ZBB  which  provided  back¬ 
ground  material  on  how  a  ZBB  process  should  be  conducted  within  an 
organization.  Next,  a  questionnaire  was  developed  which  was  aimed 
at  evaluating  the  DWMS  panel  members'  knowledge  of  ZBB  and  ob¬ 
taining  an  understanding  of  the  actual  process  the  panel  used  to  rank 
decision  packages--ZBB  as  actually  applied  in  an  organization.  The 
actual  process  as  stated  by  the  panel  members  was  compared  to  the 
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theoretical  ZBB  methodology  as  expressed  in  the  literature  in  order 
to  recommend  improvements  for  the  1983-1987  POM.  Finally,  the 
AFSC  POM  guidance  was  reviewed  in  order  to  determine  if  the  deci¬ 
sion  packages  contained  the  type  of  information  that  theoretically 
should  be  contained  in  a  decision  package. 

In  order  to  investigate  the  availability  of  AF  planning  docu¬ 
mentation,  a  literature  search  of  the  DOD's  Planning,  Programming, 
and  Budgeting  System  was  conducted  in  order  to  determine  what  docu¬ 
ments  were  available  for  use  by  the  DWMS  panel.  The  most  impor¬ 
tant  consideration  was  whether  these  documents  contained  information 
that  could  aid  in  relating  DWMS  programs  to  AF  objectives  and  needs. 
A  secondary  consideration  was  whether  these  documents  would  be 
readily  available  for  the  panel's  use  during  the  POM  process. 

Finally,  when  the  panel  members  were  interviewed,  they  were  asked 
what  documents  they  would  like  to  have  had  available,  and  the  results 
were  compared  to  the  PPBS  documents  theoretically  available. 

A  major  problem  encountered  by  the  DWMS  panel  was,  what 
is  the  best  method  for  evaluating  and  ranking  decision  packages? 

Only  a  broad  outline  was  offered  by  the  POM  guidance  with  the  speci¬ 
fic  details  left  up  to  the  individual  Mission  Area  Review  panels.  In 
order  to  collect  information  from  which  recommended  improvements 
could  be  made,  three  sources  were  used.  First,  the  panel  chairmen 
from  six  of  the  seven  Mission  Area  Review  panels  were  interviewed 
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in  order  to  discuss  the  methodology  they  used  to  evaluate  and  rank 
decision  packages .  Next,  the  DWMS  panel  members  were  interviewed 
in  order  to  obtain  their  personal  interpretation  of  the  evaluation  cri¬ 
teria  used  by  the  DWMS  and  possible  improvements  for  the  future. 
These  interviews  provided  the  practical  aspects  of  project  evaluation 
which  were  compared  to,  the  third  source  of  data,  suggested 
approaches  from  the  literature.  These  results  were  used  as  a  basis 
for  modifying  and  improving  the  scoring  model  used  by  the  DWMS 
panel  for  the  1982-1986  POM  in  order  to  increase  the  consistency  in 
project  evaluations.  Validity  of  the  model  was  checked  by  its  simi¬ 
larity  with  other  approaches  in  the  literature  and  by  the  DWMS  panel 
chairman  reviewing  and  commenting  on  the  model. 

Finally,  once  the  project  benefits  have  been  determined,  a 
model  has  to  be  developed  which  maximizes  the  expected  benefits 
from  an  R&D  project  portfolio  given  certain  resource  constraints. 

The  model  must  allow  for  project  interdependencies  rather  than 
treating  the  set  of  projects  as  independent.  The  scoring  model  that 
the  DWMS  panel  used  during  the  1982-1986  POM  did  not  consider  pro¬ 
ject  interdependencies  although  the  panel  members  did  make  trade¬ 
off  decisions  between  projects  after  the  scoring  model  produced  the 
initial  ranking.  Based  on  a  literature  search  into  R&D  project 
selection  techniques,  a  specific  model  was  chosen  and  its  use  demon¬ 
strated.  Advantages  and  disadvantages  of  the  specific  model  chosen 
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are  discussed  and  its  use  demonstrated  by  several  examples.  It  is 
important  to  note  that  the  actual  usefulness  of  the  model  to  the  DWMS 
panel  can  only  be  judged  when  it  is  applied  during  the  development  of 
the  1983-1987  ASD  POM. 


Format  of  Thesis 

Chapter  2  discusses  zero-based  budgeting  and  the  Planning, 
Programming  and  Budgeting  System  in  the  federal  government.  The 
literature  search  into  ZBB  provides  the  background  material  for  the 
investigation  into  the  problems  that  have  occurred  at  ASD  in  col¬ 
lecting  program  information  to  effectively  rank  those  programs  for 
the  ASD  POM.  The  PPBS  background  material  provides  information 
that  is  useful  in  determining  what  planning  documents  are  available 
to  aid  the  DWMS  Mission  Area  Review  panel  in  evaluating  programs 
with  respect  to  AF  goals  and  objectives. 

Chapter  3  discusses  R&cD  project  evaluation  techniques  and 
R&D  project  selection  models.  A  detailed  literature  search  into  pro¬ 
ject  evaluation  techniques  provides  the  background  information 
needed  to  develop  an  efficient  evaluation  methodology  which  the  DWMS 
panel  can  use  to  judge  the  benefits  of  the  DWMS  programs  to  the  AF. 
Different  project  selection  techniques  are  investigated  in  order  to 
determine  the  appropriate  technique  which  will  pick  the  optimal  pro¬ 
ject  portfolio. 
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Chapter  4  discusses  the  DWMS  mission  area  in  detail  and 
presents  the  panel  members'  responses  to  the  interviews.  Their 
responses  are  combined  with  the  ZBB  philosophy  in  order  to  recom¬ 
mend  changes  that  would  aid  in  improving  the  information  provided 
by  the  program  managers  about  their  programs. 

Chapter  5  develops  a  methodology  which  the  DWMS  panel 
could  use  in  the  future  to  aid  in  evaluating  DWMS  programs.  Based 
upon  the  literature  search  and  ideas  obtained  during  the  interviews, 
the  evaluation  criteria  are  improved.  Once  the  benefits  for  each  pro¬ 
ject  are  determined,  a  mathematical  programming  model  is  proposed 
which  will  optimize  the  portfolio  of  projects  rather  than  treating  each 
project  as  a  mutually  exclusive  entity. 

The  last  chapter.  Chapter  6,  presents  the  major  conclusions 
of  this  study  and  recommendations  to  the  DWMS  panel  that  can  be 
used  to  improve  the  POM  ranking  process  for  the  83-84  POM. 


Chapter  2 


THE  RESOURCE  ALLOCATION  PROCESS 
IN  THE  DEPARTMENT  OF  DEFENSE 

The  purpose  of  this  chapter  is  to  discuss  the  theoretical 
requirements  of  a  ZBB  system  and  to  discuss  the  PPBS  in  the  Depart¬ 
ment  of  Defense.  DOD’s  management,  in  the  early  1960's,  established 
a  planning,  programming,  and  budgeting  system  in  order  to  organize 
the  budget  process  and  to  tie  needs  to  programs  and  programs  to  the 
budget.  When  Jimmy  Carter  became  President  in  1977,  he  directed 
the  federal  government  to  establish  a  zero-base  budgeting  system  for 
the  federal  budget  process.  This  action  has  been  completed,  and  the 
DWMS  program  ranking  process  for  the  AF  POM  is  the  lowest  level 
of  ZBB  in  DOD.  The  theoretical  philosophy  of  ZBB  provides  a  base 
line  to  which  the  actual  DWMS  ranking  process  can  be  compared  in 
order  to  develop  improvements.  The  discussion  of  the  PPBS  in  DOD 
provides  information  pertaining  to  the  documents  that  are  available  to 
aid  the  DWMS  panel  in  the  program  evaluation  process. 

Zero-Base  Budgeting 

Although  the  concept  of  zero-base  budgeting  (ZBB)  can  be 
traced  back  to  1924,  it  was  not  until  Peter  Phyer  of  Texas  Instruments 
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published  the  results  and  experiences  of  his  firm  in  implementing 
ZBB  in  1969  that  the  process  became  widely  used.  In  its  general 
form,  ZBB  is  an  incremental  budgeting  system  that  emphasizes  incre¬ 
ments  as  well  as  decrements  to  the  budget  (McLaughlin  et  al . ,  1980: 1) . 
The  existence  of  every  program,  new  and  old,  has  to  be  justified  and 
reviewed  each  year  during  the  budget  process  in  order  to  select  those 
programs  which  provide  the  highest  possible  benefit  to  the  company 
for  the  money  expended.  If  a  new  program  generates  a  higher  return 
than  an  old  program  or  if  two  old  programs  switch  positions  with 
respect  to  their  benefit  rankings,  then  ZBB  allows  the  manager  to 
make  realistic  tradeoff  decisions  between  programs  when  some  re¬ 
quire  and  justify  higher  expenditures  and  others  require  a  decreasing 
expenditure  (Burroughs ,  1977: 1 1) . 

When  Jimmy  Carter  was  governor  of  Georgia,  he  directed 
the  state  to  convert  to  a  zero-base  budgeting  system  so  that  every 
program  could  be  reviewed  and  justified  each  year.  Each  state  agency 
would  be  required  to  identify  each  function  it  performs  and  the  per¬ 
sonnel  and  cost  to  the  taxpayers  for  performing  that  function  (Suver 
and  Brown,  1977:79).  It  was  estimated  that  for  the  state  of  Georgia's 
1972-1973  fiscal  year  approximately  11,000  decision  packages  had  to 
be  identified,  evaluated,  and  ranked  before  Governor  Carter  reviewed 
the  final  group  of  packages.  A  study  conducted  after  the  implementa¬ 
tion  of  ZBB  in  the  state  of  Georgia  indicated  that  the  process  led  to  an 
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improvement  in  the  quality  of  management  information  and  to  greater 
involvement  with  the  budget,  but  that  it  had  not  significantly  changed 
the  resources  allocated  to  programs  in  the  state  (Herzlinger,  1979:4). 

Candidate  Jimmy  Carter  promoted  ZBB  as  a  means  of  cutting 
federal  expenditures  during  his  1976  presidential  campaign.  Harvey 
states  that  the  image  of  ZBB  is  one  of  significant  cost  reduction  but 
unless  an  organization  has  already  decided  in  advance  to  cut  its  level 
of  spending,  it  is  unlikely  that  ZBB  will  produce  an  automatic  overall 
cost  reduction.  The  appropriate  place  for  ZBB  is  in  the  private  sec¬ 
tor  where  bottom-line  performance  standards  such  as  market  share, 
profit,  and  investment  return  enable  managers  to  effectively  priori¬ 
tize  their  activities.  The  value  of  ZBB  to  the  government  is  dimin¬ 
ished  by  the  relative  tack  of  useful  bottom-line  measures  of  perform¬ 
ance  (Harvey,  1978:33-34) .  Anthony  calls  zero-base  budgeting  a 
fraud  (Hyde,  1978:321),  but  Herzlinger  states  that  like  most  other 
management  techniques,  ZBB  is  neither  a  panacea  nor  a  fraud. 

It  is,  rather,  a  process  which  can  yield  substantial 
benefits  if  properly  implemented,  but  which  can  be  neutral 
or  even  pernicious  in  its  effect  if  sloppily  or  thoughtlessly 
executed  [Herzlinger,  1979:3]. 

Herzlinger  questions  whether  ZBB  is  appropriate  for  the  entire  bud¬ 
get  of  a  large  political  system.  Cheek  claims  that  the  main  benefit  is 
that  ZBB  helps  to  improve  innovation  in  an  organization. 

In  forcing  yourself  to  go  to  zero  base,  regularly  and 
systematically,  you  force  yourself  to  come  up  with  creative, 
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innovative,  alternative  ways  of  delivering  the  same  service 
at  less  cost,  or  greater  service  at  the  same  cost  [Cheek, 
1978:23]. 
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Concepts 

The  basic  concept  of  ZBB  is  that  at  regular  intervals  all  pro¬ 
gram  managers  have  to  evaluate  the  effectiveness  of  their  programs 
in  terms  of  corporate  goals  and  submit  to  their  superiors  a  detailed 
evaluation  of  their  programs  at  various  funding  levels  in  order  to 
justify  their  right  to  exist.  The  manager  must  be  able  to  justify  each 
activity's  projected  level  of  expenditures,  and  no  level  is  taken  for 
granted  (Suver  and  Brown,  1977:8  1).  The  superior  ranks  all  the  pro¬ 
grams  under  his  control  and  sends  this  priority  listing  to  the  next 
level  of  management.  Each  hierarchical  level  within  the  organization 
reviews  and  priority  ranks  all  programs  under  its  control.  By  the 
time  top  management  reviews  the  programs,  they  will  have  all  the 
inputs  by  lower  management  on  the  relative  importance  of  the  pro¬ 
grams  and  by  combining  their  judgement  with  the  views  of  lower 
management,  arrive  at  the  resource  allocation  for  the  firm. 

McCandless  discusses  six  steps  in  designing  and  imple¬ 
menting  a  zero-base  budgeting  system.  First,  top  management  has 
to  provide  all  managers  with  the  strategic  plan  of  the  organization 
and  the  specific  planning  and  policy  assumptions  used  to  derive  that 
plan.  Without  specific  guidelines,  it  would  be  impossible  to  ensure 


that  all  managers  are  using  the  same  assumptions  during  the  ZBB 
process  (McCandless,  1978:46).  Top  management  will  have  developed 


strategic  assumptions  and  goals  whether  stated  implicitly  or  explic¬ 
itly.  If  these  assumptions  can  be  communicated  explicitly  to  their 
subordinates,  then  lower  management  will  have  an  idea  where  the 
organization  is  planning  to  go  in  the  long  term  and  how  it  plans  to  get 
there.  Program  managers  will  be  able  to  formulate  their  decision 
packages  to  focus  on  and  support  the  long  range  plan  of  the  organi¬ 
zation  (Cheek,  1978:26). 

The  second  step  is  the  establishment  of  the  organization's 
decision  units,  which  are  the  smallest  discrete  units  of  activity  in 
the  organization  that  are  being  considered  for  resource  allocation. 
For  example,  in  DOD  the  decision  units  are  the  individual  program 
elements.  The  size  of  the  organization  should  be  considered  when 
designing  the  decision  units.  In  a  very  large  organization,  too  fine  of 
a  division  will  result  in  a  very  large  number  of  packages  to  evaluate 
at  superordinate  levels  because  most  decision  units  will  submit  at 
least  three  decision  packages  (McCandless,  1978:46).  The  state  of 
Georgia  had  over  11,000  packages  to  evaluate,  and  it  would  be 
impossible  for  one  person  to  evaluate  ail  the  programs  in  the  federal 
goyernment.  If  the  decision  units  are  too  large  for  analytical  or 
informational  purposes,  while  easing  the  mechanics  of  budgeting, 
managers  will  have  an  extremely  difficult  job  priority  ranking  the 
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programs.  Herzlinger  states  that  the  decision  unit  selection  problem 
can  be  minimized  by  assigning  the  decision  unit  selection  task  to  the 
managers  and  budget  people  who  will  be  using  the  data  (Herzlinger, 
1979:11). 

Next,  each  program  manager  analyzes  his  programs  and 
writes  the  decision  packages  which  justify  various  levels  of  effort  and 
resource  requirements.  Generally,  at  least  three  different  decision 
packages  are  written  for  each  program  by  the  program  manager. 

Each  package  is  a  description  of  the  program's  objectives  and  an 
evaluation  of  alternative  means  of  achieving  those  objectives  (Shapiro, 
1979:40).  The  first  package  is  the  basic  or  minimum  package,  which 
includes  those  projects  that  are  required  by  law  or  that  have  previous 
obligations  as  well  as  basic  functions  required  for  the  program  to 
exist.  It  is  the  minimum  level  below  which  the  program  ceases  to 
exist  (Suver  and  Brown,  1977:78).  Decision  package  two  represents 
the  current  funding  level  while  package  three  is  an  increment  above 
the  current  level  and  represents  an  increased  level  of  effort  (Bur¬ 
roughs,  1977;  1 1) .  McCandless  states  that  the  decision  packages  are 
usually  written  on  an  incremental  basis  where  package  two  is 
expressed  as  an  increment  above  package  one  and  accepting  package 
two  implies  the  acceptance  of  package  one  (McCandless,  1978:46). 

Burroughs  states  that  each  decision  package  should  contain 
the  following  elements:  the  purpose,  the  consequences  of  not 
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performing  that  activity,  measures  of  performance,  alternatives, 
and  cost/benefit  analysis  (Burroughs,  1977:8).  The  purpose  of  the 
decision  package  should  be  related  to  the  strategic  goals  of  the  organ¬ 
ization.  Without  this  relationship  explicitly  stated,  it  is  difficult  to 
evaluate  the  contribution  of  that  decision  package  to  the  strategic  plan 
of  the  organization.  If  the  contribution  of  a  particular  decision  pack¬ 
age  to  the  organization  was  either  not  clearly  stated  or  misstated,  it 
would  be  possible  to  overlook  a  highly  favorable  package.  A  basic 
requirement  in  order  for  any  organization  to  efficiently  achieve  its 
long  range  plan  is  for  that  organization  to  select  those  programs 
today  which  have  the  highest  contributions  to  that  plan.  Cheek  claims 
that  ".  .  .  ZBB  is  nothing  more  than  a  logical,  systematic,  disci¬ 
plined,  common  sense  framework  to  harness  and  channel  an  organi¬ 
zation's  resources  towards  preagreed  goals  and  objectives  [Cheek, 
1978:25]." 

Included  next  in  the  decision  packages  should  be  a  description 
of  the  consequences  of  not  performing  this  activity.  Cheek  claims 
that  one  of  the  major  benefits  of  ZBE  is  that  it  is  a  communications 
tool  that  improves  decision  making  and  fosters  innovation  throughout 
the  organization  (Cheek,  1978:24).  Burroughs  stages  that  one  of  the 
best  reasons  for  developing  decision  packages  is  that  "in  order  to  do 
so  effectively,  managers  must  evaluate  their  organizational  structure 
and  the  interrelationships  among  activities  and  organizational  units 
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[Burroughs,  1977:8]."  If  the  decision  packages  are  to  be  used  as  a 
valuable  communications  device  between  the  program  managers  and 
higher  management,  the  decision  packages  have  to  contain  realistic 
information  on  how  the  ultimate  goals  of  the  organization  will  be 
effected  if  this  activity  is  not  performed. 

Third,  each  decision  package  should  include  performance 
measures  for  evaluating  the  effectiveness  and  efficiency  of  the  pro¬ 
posal.  Performance  measures  are  needed  that  clearly  measure 
the  accomplishment  of  the  stated  objectives  (effectiveness)  and  assess 
these  accomplishments  in  terms  of  the  resources  used  (efficiency). 
Letzkus  states  that  explicit  articulation  of  objectives  is  required  in 
order  to  evaluate  the  effectiveness  and  efficienc,  of  a  program. 

Many  authors  feel  that  it  is  impossible  to  evaluate  the  effectiveness 
of  government  programs  in  the  absence  of  a  clear  unambiguous  state¬ 
ment  of  objectives  which  identifies  not  only  performance  standards 
but  also  expectations  over  a  specified  period  of  time.  The  output 
from  many  government  programs  is  often  difficult  to  identify  because 
it  is  in  the  form  of  intangible  services  which  results  in  a  frequent 
temptation  to  measure  resources  applied  to  the  attainment  of  the 
objectives  rather  than  to  measure  the  actual  attainment  of  those 
objectives  (Letzkus,  1978:35-36).  "Although  it  is  better  to  try  to 
quantify  the  right  performance  measures,  it  is  easier  to  select  those 
that  lend  themselves  to  quantification  [McCandless,  1978:49]." 
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People  tend  to  measure  the  easily  measurable  rather  than  what 
should  be  measured.  Nonetheless,  a  basic  requirement  of  any  pro¬ 
gram  evaluation  is  a  clear,  specific  statement  of  objectives  (Letzkus, 
1978:35). 

The  fourth  element  in  the  decision  package  should  be  a  dis¬ 
cussion  of  the  alternatives  evaluated  by  the  program  manager 
(Burroughs,  1977:8).  Cheek  claims  that  ZBB  increases  innovation  in 
an  organization  by  forcing  program  managers  to  think  creatively  in 
coming  up  with  alternative  ways  of  operating  (Cheek,  1978:25).  The 
program  manager  should  evaluate  different  ways  of  and  different 
levels  of  effort  for  performing  the  same  function  (Burroughs,  1977:8). 
Included  in  the  alternative  means  of  achieving  the  stated  objectives 
should  be  the  reasons  why  certain  alternatives  were  rejected 
(McCandless,  1978:45).  One  of  the  most  significant  features  of  ZBB 
is  that  it  communicates  to  upper  management  what  the  people  respon¬ 
sible  for  the  programs  consider  as  areas  of  importance  and  where 
the  program  managers  would  reallocate  resources  given  either  an 
increment  or  decrement  in  their  budget. 

Finally,  the  last  element  in  the  decision  package  is  to  pro¬ 
vide  a  description  of  the  costs  and  expected  benefits  (Burroughs, 
1977:8).  Costs,  although  difficult  to  estimate  especially  for  uncer¬ 
tain  R&fD  programs,  do  not  present  as  difficult  a  problem  as  the 
estimation  of  future  benefits  especially  for  federal  programs  when 
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the  output  is  in  the  form  of  intangible  services  and  cannot  be  well 
defined.  How  does  one  measure  this  output  or  expected  benefits? 
Letzkus  discusses  the  problems  involved  in  choosing  the  appropriate 
indicator  of  output,  whether  single,  multiple,  or  composite  measures 
are  appropriate,  in  order  to  determine  the  benefits  of  a  particular 
program  (Letzkus,  1978:35). 

The  fourth  step  of  the  ZBB  process  is  that  the  decision  pack¬ 
ages  are  ranked  according  to  their  contribution  to  the  goals  of  the 
organization  (McCandless,  1978:46).  The  difference  between  ZBB 
and  other  budgeting  approaches  is  that  with  ZBB  every  manager  in 
the  company's  hierarchy  reviews  and  prioritizes  all  programs  under 
his  control.  Burroughs  states  that  "the  process  identifies,  to  all 
levels  of  management,  the  costs,  benefits,  and  suggested  operational 
levels  associated  to  reach  their  objectives  [Burroughs,  1977:8]." 
Shapiro  suggests  that  the  ranking  process  be  conducted  by  a  group 
which  evaluates  incremental  decision  packages  against  one  another 
(Shapiro,  1979:42) .  Suver  and  Brown  suggest  that  the  group  members 
should  be  the  program  managers  whose  packages  are  being  ranked 
and  their  supervisor  should  be  the  panel  chairman  (Suver  and  Brown, 
1977:78).  McLaughlin  et  al.  discusses  the  use  of  and  problems  asso¬ 
ciated  with  the  use  of  expert  panels,  nominal  group  techniques,  and 
scoring  models  to  rank  programs  (McLaughlin  et  al.,  1980:4).  The 
U.S.  Office  of  Management  and  Budget  Bulletin  No.  77-9  states: 
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The  ranking  shows  the  relative  priority  that  discrete  1 

increments  of  service  or  other  outputs  have  in  relation  to  | 

other  increments  of  service  or  other  outputs.  The  process 
is  explicitly  designed  to  allow  higher  level  managers  the 
opportunity  to  bring  their  broader  perspectives  to  bear  on 
program  activities  by  allowing  them  to  rank  the  decision 
packages  and  make  program  trade-offs  [Hyde,  1978:311], 

McCandless  states  that  the  fifth  step  is  the  preparation  of 
the  final  budget  in  a  contingency  form  (McCandless,  1978:46).  Top 
management  reviews  their  final  priority  ranking  of  programs  and 
compares  the  cumulative  cost  of  those  programs  to  the  total  budget 
figure.  All  decision  packages  for  which  the  cumulative  total  is  less 
than  the  budgeted  amount  are  included  in  the  portfolio  of  programs 
while  those  whose  cumulative  total  are  greater  than  the  budgeted 
amount  are  excluded.  All  programs  which  are  included  in  the  final 
portfolio  constitute  the  operating  budget  for  the  organization  (Shapiro, 

1979:40).  The  purpose  of  the  ZBB  priority  ranking  is  to  indicate 
which  decision  packages  should  be  added  first  if  the  budget  is  in¬ 
creased  and  which  should  be  the  first  to  go  if  the  budget  is  decreased 
(McCandless,  1978:46).  The  decision  packages  deleted  or  added  will 
be  based  upon  the  priorities  established  by  those  managers  responsi¬ 
ble  for  the  execution  of  those  decision  packages  (Burroughs,  1977:8). 

The  final  step  is  that  the  managers  are  given  feedback  on  the 
final  rankings.  McCandless  states  that  "...  the  reasons  for  any 
changes  in  rankings  made  by  higher  management  levels  are  passed 
down  to  the  lower-level  managers.  This  should  help  build  up  trust 
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in  senior  management  [McCandless,  1978:46]."  Since  one  of  the  pur¬ 
poses  of  the  ZBB  process  is  to  allow  higher  management  to  make 
realistic  tradeoffs  between  programs  and  to  redirect  funds  based 
upon  the  recommendations  and  evaluations  of  lower  managers,  feed¬ 
back  should  be  provided  so  that  lower  managers  don't  feel  "done  to" 
if  their  program  is  cancelled  or  don't  feel  like  the  process  is  a  waste 
of  time  (Burroughs  ,  1977: 1 1) .  If  feedback  is  provided,  organizational 
trust  in  top  management  will  be  increased  because  subordinates  will 
trust  their  superior's  ability  and  fairness  and  will  submit  reliable, 
complete,  and  unbias  information  in  the  future  (McCandless,  1978:47). 

Benefits 

The  major  benefits  of  the  zero-base  budgeting  process  are 
that  the  process  fills  a  communications  void  in  the  organization, 
allows  top  management  to  make  realistic  tradeoff  decisions  between 
programs,  allows  managers  to  obtain  a  better  understanding  of  how 
their  organization  is  structured  and  operates,  and  allows  lower  level 
managers  to  feel  an  ownership  for  the  final  budget  because  of  their 
contributions  to  the  process.  Harvey  claims  that  ZBB  is  a  useful 
general  management  process  because  it  fills  a  communications  void 
in  the  organization  (Harvey,  1978:33).  One  of  the  most  significant 
characteristics  of  ZBB  is  that  information  flows  from  the  bottom  of 


the  organization  to  the  top  (Burroughs,  1977:11).  The  ZBB  process 


is  an  invaluable  tool  to  communicating  ideas  and  legitimate  needs  up¬ 
wards  to  key  decision  makers  (Cheek,  1978:24).  Top  management,  by 
reviewing  the  information  contained  in  the  decision  packages  and  the 
priority  ranking  established  by  lower  managers,  can  make  a  more 
realistic  redirection  of  funds  from  programs  that  have  a  decreasing 
resource  requirement  to  programs  which  require  a  higher  resource 
requirement.  This  reallocation  of  funds  is  in  line  with  not  only  top 
management’s  priorities  but  also  utilizes  the  priorities  established 
by  subordinates  (Burroughs,  1977:8). 

Another  important  aspect  of  ZBB  is  that  in  establishing  and 
justifying  each  decision  package ,  all  managers  are  able  to  obtain  a 
clearer  picture  about  their  organization.  "...  perhaps  the  most 
useful  aspect  of  a  zero-base  system  is  a  complete  identification  of 
the  current  and  proposed  programs  and  their  costs  [Suver  and  Brown, 
1977:84]."  It  forces  managers  to  explicitly  link  their  resource 
requests  with  the  benefits  of  their  proposed  activities  and  the  objec¬ 
tives  of  the  organization  (Shapiro,  1979:40).  In  organizing  and  col¬ 
lecting  the  information  required  for  the  decision  packages,  the 
program  manager  can  obtain  a  clearer  picture  of  the  interrelation¬ 
ships  within  his  program,  and  the  alternative  means  available  to 
accomplish  the  program's  objectives. 

Decision  packages  therefore  offer  top  management, 
boards  of  directors,  or  finance  committees  the  opportunity 


to  gain  a  better  understanding  of  what's  going  on  throughout 
the  organization,  what  the  objectives  of  the  various  subunits 
are,  and  how  they  are  to  be  achieved  [McCandless,  1978:46], 

Harvey  claims  that  an  additional  benefit  is  that  all  managers 
throughout  the  organization's  hierarchy  feel  an  "ownership"  for  the 
final  budget  because  of  their  personal  participation  in  the  ranking 
process.  For  many  managers,  the  implementation  of  a  ZBB  process 
is  the  first  real  opportunity  that  they  have  to  significantly  influence 
the  direction  of  the  organization. 

The  value  of  budget  ownership  is  seen  when  funds  are 
finally  allocated  to  the  ranking  list.  Budget  managers  do  not 
feel  'done  to'  but  rather  acknowledge  the  implementation  of 
what  was,  in  effect,  their  own  budget  and  plan  [Harvey,  1978:34], 

Problems 

Many  authors  have  discussed  the  problems  with  zero-based 
budgeting.  Anthony  says  that  zero-based  budgeting  is  a  fraud.  He 
claims  that  the  name  implies  that  the  program  manager  starts  each 
budget  year  at  zero  and  has  to  justify  every  dollar  requested  when  in 
fact  a  level  of  approximately  70  to  80  percent  of  the  current  spending 
level  is  established  as  the  minimum  package  and  given  a  cursory 
examination  while  attention  is  focused  on  the  incremental  packages. 

In  a  large  organization,  the  large  number  of  required  decision  pack¬ 
ages  would  make  the  process  unmanageable.  A  thorough  analysis 
during  the  time  available  in  the  budget  process  would  be  impossible. 
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Another  problem  is  that  program  rankings  will  be  influenced  by  the 
expected  amount  of  funds  available  or  will  be  deliberately  structured 
so  that  essential  or  politically  popular  packages  are  given  a  low 
ranking.  If  these  lower  ranked  packages  are  reinstated,  all  of  the 
higher  ranked  packages  will,  also,  be  included  in  the  budget  (Hyde, 
1978:322). 

Saver  and  Brown  cited  three  disadvantages  to  the  ZBB  pro¬ 
cess  from  the  experiences  in  the  state  of  Georgia. 

1)  An  increase  in  the  time  and  effort  required  to  prepare 
the  budget. 

2)  When  viewing  the  end  result  of  the  old  budgeting  system 
and  ZBB,  ZBB  has  not  significantly  affected  the  allocation  of  funds  to 
programs. 

3)  The  decision  package  ranking  approach  was  ineffective  in 
meeting  changes  in  funding  levels  (when  funds  were  increased  or 
decreased,  new  decision  packages  were  created  instead  of  using  the 
initial  decision  package  rankings)  (Suver  and  Brown,  1977:80). 

Further,  McLaughlin  et  al.  noted  that  additional  problems  could  arise 
if  the  process  is  not  done  in  good  faith  or  if  there  is  a  great  amount  of 
post-acceptance  manipulations  (McLaughlin  et  al.  ,  1980:2). 

As  discussed  earlier,  one  of  the  major  problems  associated 
with  the  evaluation  of  any  public  program  is  how  does  one  measure  the 
outputs  of  such  a  program?  Suver  and  Brown  discuss  the  problem 
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that  output  measurements  in  terms  of  achievement  are  either  lacking, 
too  subjective,  or  too  argumentative  (Suver  and  Brown,  1977:81). 
Letzkus  addresses  the  performance  measurement  problem  in  terms 
of  two  factors: 

1)  differentiation  between  measures  of  work  accomplished 
and  the  effectiveness  of  goal  accomplishment,  and 

2)  determination  as  to  whether  single,  multiple,  or  compo¬ 
site  measures  will  provide  a  true  measure  of  performance  (Letzkus, 
1978:36). 

Suver  and  Brown  go  on  to  say  that  problems  can  occur  when  manage¬ 
ment  tries  to  compare  outputs  of  non-related  decision  packages  which 
is  required  in  the  ZBB  process  (Suver  and  Brown,  1977:81). 

Another  problem  is  that  the  implementation  of  the  ZBB  pro¬ 
cess  represents  major  changes  for  an  organization.  McCandless 
discusses  organizational  change  in  terms  of  the  model  developed  by 
Kurt  Levin  where  one  must  unfreeze  the  old  attitudes,  change  them, 
and  refreeze  the  new  attitudes  in  order  to  achieve  the  desired  state. 

In  order  for  the  refreezing  state  to  be  effective,  the  organization's 
reward  system  has  to  reinforce  the  new  behaviors  (McCandless, 
1978:47).  Cheek  claims  that  no  matter  how  good  a  new  idea  is, 
large  organizations  resist  change  (Cheek,  1978:24) .  Suver  and  Brown 
state  that  many  people  feel  that  the  replacement  of  an  old  system 
with  a  new  system  implies  that  the  old  system  was  inadequate.  Many 
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people  in  organizations  resent  and  resist  change  (Suver  and  Brown, 
1977:81). 

The  problems  associated  with  the  behavior  implications  of 
ZBB  are  a  major  factor  in  the  acceptance  of  the  process  in  an 
organization.  ZBB  requires  every  program  to  be  reviewed  once  a 
year  and  its  right  to  exist  justified.  This  "life  or  death"  situation 
can  be  perceived  as  a  threat  by  the  program  managers  and  other  em¬ 
ployees  working  on  the  program  (Letzkus,  1978:40).  They  will  have  a 
tendency  to  adjust  the  information  provided  in  the  decision  packages 
in  order  to  increase  the  likelihood  of  the  acceptance  of  their  program. 

Finally,  McLaughlin  et  al.  discuss  that  perhaps,  because  of 
the  amount  of  work  involved,  the  ZBB  process  could  better  be 
addressed  during  slack  periods  of  the  budget  cycle.  The  review  of 
the  budget  may  not  be  optimally  performed  during  a  period  of  heavy 
workload  and  deadlines  (McLaughlin  et  al.  ,  1980:2).  Anthony  suggests 
that  perhaps  the  zero-base  analysis  could  more  effectively  take  place 
outside  of  the  budget  cycle  (Hyde,  1978:322). 

This  section  presented  the  concepts,  benefits,  and  problems 
with  zero-base  budgeting  which  has  now  been  fully  integrated  into  the 
federal  government's  budgeting  process.  The  next  section  discusses 
PPBS  in  the  Department  of  Defense  starting  with  the  planning  phase 
during  which  strategies  and  long  range  plans  are  developed,  contin¬ 
uing  through  the  programming  phase  where  programs  are  developed 
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to  fulfill  the  long  range  plans,  and  finally,  ending  with  the  budgeting 
phase  where  the  financial  plan  to  fund  the  desired  programs  is 
developed.  Especially  interesting  to  the  DWMS  panel  members 
would  be  those  documents  which  update  the  FYDP,  provide  feedback 
about  the  previous  POM,  and  relate  programs  to  AF  objectives. 

Planning,  Programming,  and  Budgeting 

During  the  1950s,  the  only  control  that  the  Office  of  the 
Secretary  of  Defense  (OSD)  exercised  on  the  annual  Air  Force  (AF) 
program  was  fiscal  control.  Essentially,  there  was  no  long  range 
planning  as  the  AF  looked  only  at  one  year  in  the  future  when  budget¬ 
ing  and  there  was  very  little  integrated  inter- service  planning. 
Within  the  AF,  planning  in  response  to  threat  analysis  was  not  con¬ 
nected  to  budgeting.  A  Planning,  Programming,  and  Budgeting 
System  (PPBS)  was  established  within  the  Department  of  Defense 
(DOD)  by  Charles  Hitch,  the  OSD  Comptroller  under  Secretary  of 
Defense  (SecDef)  McNamara,  in  order  to  correct  these  deficiencies 
in  the  DOD  resource  allocation  process.  The  purpose  of  the  PPBS 
is  stated  below: 

To  bridge  the  gap  between  planning  and  budgeting,  we 
developed  a  new  programming  system  which  relates  missions 
to  forces,  forces  to  programs,  and  programs  to  budgets,  and 
the  whole  is  projected  at  least  five  years  ahead  [Fairfield, 
1979:51]. 


Perhaps  the  greatest  benefit  from  the  PPBS  is  that  it  insures  that 


budget  dollars  are  related  to  programs  (Fairfield,  1979:5  1).  PPBS  is 

very  important  to  the  AF  in  the  allocation  of  resources  to  programs. 

The  PPBS  is  the  single  instrumentality  for  allowing 
resource  allocation  decisions  to  be  made  and  recorded. 
Consequently,  any  successful  attempts  to  influence  the  allo¬ 
cation  of  resources  will  ultimately  be  reflected  in  this  core 
planning  system  [AFP  172-4,  1979:13]. 

The  PPBS  cycle  is  composed  of  three  separate  phases;  the 

planning  phase,  the  programming  phase,  and  the  budgeting  phase, 

all  of  which  are  based  on  the  threat  analysis  by  the  Joint  Chiefs  of 

Staff  (JCS).  The  process  culminates  in  the  President's  budget  being 

submitted  to  Congress  in  January.  All  three  phases  are  a  continuing 

process  due  to  the  overlap  in  budget  cycles.  One  complete  cycle 

from  the  issuance  of  the  Joint  Intelligence  Estimate  for  Planning 

(JIEP)  by  the  JCS  to  the  passage  of  the  budget  by  Congress  is  approx- 

\ 

imately  a  33  month  process .  During  this  process,  communication 
continually  occurs  between  the  President,  the  SecDef,  the  JCS,  the 

Air  Staff,  and  the  user  commands.  The  purpose  is  to  determine  the 

/ 

most  efficient  allocation  of  budget  resources  in  order  to  meet  current 
and  projected  threats.  The  PPBS  can  be  summarized  as  follows: 
based  on  the  anticipated  threat,  a  strategy  is  developed;  in  support  of 
that  strategy,  force  requirements  are  developed;  based  on  these 
requirements,  programs  are  developed  to  provide,  on  an  orderly 
basis,  military  resources  over  a  period  of  time,  with  due  consider¬ 
ation  of  the  total  cost  to  the  nation;  and  lastly,  funds  must  be 
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budgeted  in  such  a  manner  as  to  obtain  the  required  forces  and  wea¬ 
pon  systems  within  the  resources  that  the  nation  provides  (General 
Dynamics,  1979:4).  It  "fundamentally  attempts  to  set  the  balance 
among  the  complimentary  force  and  capability  characteristics  of 
structure,  readiness,  and  modernization  [AFP  172-4,  1979:19]." 

Planning  Phase 

The  purpose  of  the  planning  phase  is  to  continually  analyze 
the  current  and  projected  threats,  determine  how  to  best  meet  those 
threats,  determine  what  the  current  and  projected  capabilities  are  in 
meeting  those  threats,  calculate  current  and  expected  deficiencies, 
establish  priorities ,  and  to  determine  the  best  resource  allocation  in 
order  to  meet  national  objectives.  Three  somewhat  independent 
planning  systems  provide  input  to  and  are  tied  together  by  the  PPBS 
and  are  aimed  at  providing  the  unified  and  specified  commanders  with 
the  forces  they  need.  They  are  the  Development  Planning  System, 
the  Joint  Operational  Planning  System,  and  the  Joint  Strategic  Plan¬ 
ning  System  (AFP  172-4,  1979:16).  Figure  1  shows  the  relationship 
between  the  planning  systems  and  the  feedback  from  the  operational 
planning  system  to  the  other  systems. 

Development  Planning  System 
The  development  planning  system  is  concerned  with  the 
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Figure  1:  PPBS  Planning  System  (AFP  172-4,  1979:16) 

qualitative  improvement  of  the  military  force  structure  and  focuses 
on  the  modernization  of  capability.  Developmental  activity  can  be 
defined  as  Research,  Development,  Test  and  Evaluation,  and  Acqui¬ 
sition  with  the  majority  of  the  resources  expended  on  major  weapon 
systems  acquisition.  The  developmental  cycle  is  not  tied  directly 
to  any  given  PPBS  cycle  but  rather  the  monetary  implications  of 
various  program  alternatives  are  evaluated  at  key  decision  points  or 
milestones.  Figure  2.  The  funding  profiles  not  only  depend  upon  an 

economically  sound  program  but  also  are  adjusted  yearly  in  response 

\ 

to  the  changing  world  environment  and  priorities  among  Air  Force 
programs  (AFP  172-4,  1979:19). 
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Figure  2:  Major  Systems  Acquisition  Cycle 
(AFP  172-4,  1979:19) 


Joint  Strategic  Planning  System 

The  Joint  Chiefs  of  Staff  (JCS)  have  the  responsibility  of 
formulating  the  strategic  plans  for  the  United  States  to  meet  the 
challenges  of  actual  or  potential  enemy  threats.  The  process  through 
which  the  JCS  performs  this  function  is  the  Joint  Strategic  Planning 
System  (JSPS)  which  eventually  supplies  planning  and  programming 
information  for  the  PPBS  (Fairfield,  1979:54).  The  PPBS  is  the  pro¬ 
cess  which  ties  the  threat  assessment  of  the  JCS  and  other  military 
planners  to  the  budget  the  President  presents  to  Congress  in  January. 
Figure  3  shows  the  relationship  between  the  JSPS  and  the  PPBS.  In 
terms  of  the  influence  on  the  resource  allocation  process,  of  all  the 
documents  prepared  by  the  JCS,  the  Joint  Strategic  Planning  Docu¬ 
ment  (JSPD)  and  the  Joint  Program  Assessment  Memorandum 
(JPAM)  are  the  most  influential  in  the  PPBS  process  (AFP  172-4, 
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1979:17). 


The  JSPS  starts  with  an  analysis  of  the  threat  to  the  United 
States  and  its  allies  and  the  results  are  presented  in  the  Joint  Intelli¬ 
gence  Estimate  for  Planning  (JIEP).  Information  is  provided  by  the 
services,  the  Defense  Intelligence  Agency  (DIA),  and  from  within  the 
JCS.  The  JIEP  assesses  the  threat  over  a  short  to  mid-range  time 
period  and  provides  information  that  forms  the  basis  for  a  JCS  inter¬ 
nal  working  document  called  the  Joint  Strategic  Planning  Document 
Supporting  Analysis  (JSPDSA)  which  is  published  in  two  parts  (Fair- 
field,  1979:54). 

Part  1  of  the  JSPDSA  addresses  the  JCS's  ideas  on  strategy 
and  provides  force  planning  guidance.  This  document  provides  other 
military  planners  with  the  JCS  perception  of  the  required  military 
strategy  in  order  to  meet  any  perceived  threat.  The  JSPDSA  Part  1 
is  used  by  the  Air  Staff  in  order  to  conduct  the  USAF  Objective  Force 
exercise  (Fairfield,  1979:54, 56). 

Prior  to  the  USAF  Objective  Force  exercise,  the  component 
commanders  submit  to  the  Air  Staff  their  "minimum  risk  forces." 
These  forces  constitute  what  the  commanders  think  are  required  to 
defeat  the  threat  as  presented  by  the  JCS,  and  these  estimates  are 
made  without  considering  the  constraints  that  will  be  applied  later  by 
the  PPBS.  The  purpose  of  the  Objective  Force  exercise  is  to  take 
the  minimum  risk  force  structure  and  evaluate  the  possibility  of 
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Figure  3  (cont.):  PPRS  and  JSPS 


obtaining  these  forces  within  the  current  and  projected  U.S.  industrial 
capacity.  The  final  result  of  the  exercise  is  the  "prudent  risk  force" 
which  forms  the  basis  for  the  JSPDSA  Part  2  and  the  AF  Program 
Objective  Memoranda  (POM1  exercise  (Fairfield,  1979:56) . 

The  JCS,  after  reviewing  the  prudent  risk  force  outlined  in 
Objective  Force,  publishes  the  JSPDSA  Part  2  which  describes  the 
force  structure  needed  to  allow  military  leaders  to  carry  out  the 
national  strategy  with  a  reasonable  degree  of  success.  In  deriving 
the  required  force  structure,  constraints  such  as  budget  limitations, 
manpower  reserve,  material  availability,  industrial  capability,  and 
technology  are  considered.  If  the  prudent  risk  force  cannot  be 
attained  within  the  next  five  to  seven  years,  the  JCS  must  provide 
advice  on  the  actions  necessary  to  meet  those  deficiencies  (Fairfield, 
1979:56). 

Based  upon  the  previous  studies  conducted  by  the  JCS,  the 
Air  Staff,  and  the  major  commands,  the  JCS,  in  fulfilling  their  obli¬ 
gation  to  provide  advice  to  the  National  Command  Authorities, 
publishes  the  Joint  Strategic  Planning  Document  (Fairfield,  1979:54). 
This  document  provides  the  advice  of  the  JCS  to  the  President,  Na¬ 
tional  Security  Council,  and  the  SecDef  on  the  military  strategy  and 
force  structure  required  to  attain  the  national  security  objectives  of 
the  U.S.  Published  in  November  this  document's  purpose  is  to 
strongly  influence  the  forthcoming  Consolidated  Guidance  that  will  be 
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issued  by  the  Secretary  of  Defense  (AFP  172-4,  1979:18).  Contained 


in  this  document  are: 

1)  Comprehensive  appraisal  of  the  military  threat  to  the 
United  States , 

2)  A  statement  of  recommended  military  objectives, 

3)  The  recommended  military  strategy  to  obtain  the  ob- 

j  ectives , 

4)  Summary  of  the  JCS  planning  force  levels  that  could, 
with  reasonable  assurance  of  success,  execute  the  military  strategy, 

5)  JCS  views  on  the  attainability  of  the  recommended  force 
levels  within  fiscal  constraints,  manpower  resources,  material 
availability,  technology,  and  industrial  capacity,  and 

6)  The  initial  appraisal  of  the  risk  associated  with  the  pro¬ 
grammed  force  levels  (Command  Magazine,  1979:8). 

The  purpose  of  this  document  is  to  provide  the  comprehensive  recom¬ 
mendations  of  the  JCS. 

The  planning  cycle  is  completed  the  March  after  the  issuance 
of  the  JSPD  with  the  publication  of  the  Consolidated  Guidance  (CGI  by 
the  Secretary  of  Defense,  This  document,  issued  after  communica¬ 
tions  with  the  President,  the  JCS,  and  the  Air  Staff,  presents  the 
Secretary  of  Defense's  views,  interpretations,  and  priorities  of  the 
force  structure  recommended  in  the  JSPD.  The  CG  provides  specific 
guidance  in  such  areas  as  nuclear  forces,  general  purpose  forces, 
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logistics,  manpower,  research  and  development,  telecommunica¬ 
tions,  and  intelligence  and  provides  the  rational  for  the  defense  pro¬ 
gram  (AFP  172-4,1979:13).  Not  only  is  the  total  obligation  authority 
by  service  established  in  this  document  but  also  any  other  constraints 
that  the  SecDef  wishes  to  impose.  The  CG  provides  the  final  guidance 
to  the  services  for  the  preparation  of  the  POM. 

Mission  Area  Analysis 

The  aim  of  the  mission  area  analysis  (MAA)  is  to  provide  a 
basic  understanding  about  what  the  Air  Force  needs  to  do  in  order  to 
perform  the  missions  assigned  to  it  (AFP  172-4,  1979:20),  Com¬ 
peting  with  other  federal  agencies  for  the  available  funds,  the  Air 
Force  has  to  allocate  limited  resources  in  a  multi-mission  environ¬ 
ment  in  trying  to  maximize  combat  capability.  The  MAA  begins  with 
the  identified  objectives  listed  in  the  JSPDSA  Part  1  and  ends  with  an 
output  which  shows  how  these  objectives  can  be  met.  This  process 
will  aid  decision  makers  at  all  levels  to 

1)  recognize  needs, 

2)  prioritize  needs  within  mission  areas,  and 

3)  compare  alternative  solutions  in  terms  of  the  marginal 
return  each  makes  to  combat  capability  (Fairfield,  1979:58). 

MAA  provides  a  mechanism  for  identifying  the  most  critical 
aspects  of  the  Air  Force's  mission  and  relates  these  aspects  to  the 
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desired  combat  output  and  Air  Force  objectives.  The  results  of  the 
MAA  are  published  in  the  Air  Force  Planning  Guide  (AFPR)  which 
provides  the  criteria  by  which  resource  allocation  alternatives  can  be 
assessed.  The  AFPG  can  be  very  useful  during  the  POM  development 
because  it  documents  what  the  AF  needs  to  do  to  meet  current  and 
projected  deficiencies  and  provides  a  prioritized  needs  list  which 
results  from  the  MAA  (AFP  172-4,  1979:20). 

Programming  Phase 

The  programming  phase  begins  with  the  issuance  of  the 
Consolidated  Guidance  in  March,  continues  through  the  POM  exercise 
and  submission,  and  ends  with  the  SecDef  issuing  the  Amended  Pro¬ 
gram  Decision  Memoranda  (PDM)  in  August  (AFP  172-4,  1979:13), 

The  Air  Force  Budget  states  that  the  main  purpose  of  USAF  program¬ 
ming  is  ".  .  ,  to  schedule  the  application  of  current  and  projected 
USAF  resources  to  the  requirements  of  an  orderly  progression  of 
aerospace  capabilities  essential  to  National  Security  Objectives  [AFP 
172-4,1979:21]."  USAF  programming  attempts  to  provide  the  mini¬ 
mum  capabilities  required  to  counter  a  perceived  threat  within  the 
given  resource  constraints.  The  purpose  of  the  planning  phase  is  to 
convert  Air  Force  requirements  stated  as  needs  and  deficiencies  to 
specific  budget  requests  so  that  the  President  can  submit  his  budget 
proposal  to  Congress. 
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During  the  programming  phase,  the  SecDef  provides  the  J CS 
and  the  Air  Staff  with  the  Consolidated  Guidance  which  describes  the 
SecDef' s  opinions  and  priorities  for  resource  allocation  in  the  Air 
Force.  Based  upon  the  CG,  the  Air  Staff  prepares  the  Air  Force's 
POM  which  .  .  identifies  resources  required  to  perform  assigned 
missions  and  distribution  of  resources  to  set  up  a  balanced  program 
consistent  with  fixed  constraints  established  by  DOD,  OMB,  Congress, 
ecc.  [ASD  Pamphlet  800-19,  1979:49]."  The  JCS  analyzes  both  the  CG 
and  the  POMs  submitted  by  the  services  and  provides  the  SecDef  with 
the  Joint  Program  Assessment  Memorandum  (JPAM).  The  JPAM  is 
the  JCS's  assessment  of  the  capabilities  of  the  POM  force  recom¬ 
mendations  in  meeting  perceived  threats. 

Five  Year  Defense  Plan 

The  Five  Year  Defense  Plan  (FYDP)  is  the  official  program 
summarizing  the  Secretary  of  Defense  approved  plans  and  programs 
for  the  Department  of  Defense  (AFP  172-4,  1979:22).  The  FYDP 
details  costs  and  manpower  requirements  for  the  budget  year  and  the 
budget  year  +  1  through  budget  year  +  4.  Also,  included  for  reference 
are  the  current  and  previous  year  requirements.  It  is  updated  three 
times  a  year:  when  the  services  submit  their  POM  in  May,  when  the 
budget  submission  is  sent  to  the  Office  of  the  Secretary  of  Defense 

(OSD)  for  review  in  September,  and  when  the  President  submits  the 
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budget  to  Congress  in  January  (Fairfield,  1979:61). 


The  FYDP  consists  of  both  force-oriented  programs  and 
support-oriented  programs  and  is  continually  being  reviewed  and 
modified  in  order  to  improve  resource  allocation  to  force-oriented 
mission  programs  consistent  with  DOD  management  needs  and  direc¬ 
tion.  Ten  major  force  programs  divide  the  FYDP  as  shown  in  Figure 
4. 


Program  1 

- 

Strategic  Forces 

Program  2 

- 

General  Purpose  Forces 

Program  3 

- 

Intelligence  and  Communications 

Program  4 

- 

Airlift/Sealift 

Program  5 

- 

Guard  and  Reserve  Forces 

Program  6 

- 

Research  and  Development 

Program  7 

- 

Central  Supply  and  Maintenance 

Program  8 

- 

Training,  Medical  and  Other  General 
Personnel  Activities 

Program  9 

- 

Administration  and  Associated  Activities 

Program  0 

- 

Support  of  Other  Nations 

Figure  4:  Major  Force  Programs  (AFP  172-4,  1979:23) 


A  program  is  defined  as  an  aggregation  of  program  elements 
which  not  only  reflects  a  force  mission  or  a  support  mission  of  DOD 


but  also  contains  the  resources  needed  to  achieve  an  objective  or  plan. 
A  program  element  describes  a  specific  mission  in  terms  of  organi¬ 
zational  ntities  and  resources  such  as  forces,  manpower,  material, 
and  cost  required  to  perform  the  mission  and  is  the  primary  data 
element  in  the  FYDP  (Fairfield,  1979:60). 

Program  Objective  Memorandum 

"The  POM  is  used  at  the  requesting  level  to  project  five 
year's  requirements  for  dollars  by  mission  or  weapon  system  to  up¬ 
date  the  Five  Year  Defense  Plan  (FYDP)  [ASD  Pamphlet  800-  19, 
1979:49]."  The  POM  is  based  upon  the  guidance  issued  in  the  JSPDA2 
and  the  CG  and  provides  the  detailed  force  structure  and  level  of 
activity  required  to  fulfill  AF  objectives  within  anticipated  funding 
levels.  Essentially,  the  process  is  one  of  fitting  the  next  five  year's 
forces  within  a  fiscally  constrained  box  (AFP  172-4,  1979:31). 

Currently,  the  POM  process  starts  the  zero-base  budgeting 
process  in  DOD  for  the  federal  government.  Each  program's 
description  for  the  POM  evaluation  is  written  in  the  appropriate 
decision  package  format.  Not  only  does  the  Air  Force  have  to  devise 
a  force  structure  in  the  POM  that  will  be  able  to  meet  any  perceived 
threats  but  also  the  AF  has  to  rank  all  the  decision  packages  according 
to  the  rules  of  zero-base  budgeting.  The  POM  is  related  to  the  AF's 
budget  in  that  the  first  year  of  the  POM  is  the  requested  budget  for 
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the  Air  Force  (AFP  172-4,  1979:31). 

Feedback  from  the  Air  Staff's  POM  exercise  is  published  in 
the  Air  Force  P- series  documents.  These  documents  not  only  sum¬ 
marize  the  POM  force  structure  but  also  capsulize  the  POM  guidance 
items  as  they  have  changed  from  the  last  FYDP  position.  The  P- 
series  documents  are  the  only  formal  system  within  the  AF  established 
in  order  to  keep  commanders  informed  of  changes  in  the  FYDP 
(Fairfield,  1979:67). 

Joint  Program  Assessment  Memorandum 

Once  the  services  have  completed  their  POM  requests,  the 
JCS  provide  their  input  into  the  programming  process  by  reviewing 
the  POMs  and  providing  their  analysis  in  the  Joint  Program  Assess¬ 
ment  Memorandum  (JPAM) .  This  document  contains  the  advice  of 
the  JCS  to  the  SecDef  for  use  in  reviewing  the  POM,  developing  budget 
guidance,  and  deciding  on  specific  programs  (Command  Magazine, 
1979:7).  The  SecDef  reviews  the  POM  and  the  JPAM,  and  by  com¬ 
municating  with  the  President,  the  JCS,  and  the  Air  Staff,  the  SecDef 
formulates  the  Program  Decision  Memoranda  (PDMs)  in  August. 

Program  Decision  Memoranda 

The  programming  phase  is  considered  completed  when  the 
Program  Decision  Memoranda  (PDMs)  or,  if  required,  the  Amended 
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Program  Decision  Memoranda,  which  represents  the  SecDef's  basic 
program  decisions,  is  issued  in  August  (AFP  172-4,  1979:13).  It 
incorporates  all  the  program  decisions  through  the  SecDef's  level 
and  forms  the  basis  for  the  budget  proposal.  From  a  budgeter's 
point  of  view,  the  PDM/APDM  represents  the  approved  resource 
(budget)  levels  reflected  in  the  FYDP  (Command  Magazine,  1979:7). 


Budgeting  Phase 


The  budgeting  phase  begins  with  the  issuance  of  the  Budget 
Guidance  in  September  and  is  completed  when  the  President  presents 
his  budget  in  January.  Budgeting  as  defined  in  ASD  Pamphlet  800-  19 
is  .  .  an  allocation  of  programmed  costs  over  time,  the  acquisition 
having  been  approved  [ASD  Pamphlet  800-19,  1979:49].” 

The  budget  is  the  culmination  of  the  Planning,  Program¬ 
ming,  and  Budgeting  System .  First,  we  plan,  then  having 
established  objectives  and  goals,  we  program:  fit  resources 
to  those  plans.  The  last  action  is  to  budget:  to  calculate  in 
detail  the  resources  needed  and  available  for  our  program 
[AFP  172-4,  1979:31]. 

After  the  Budget  Guidance  is  issued,  the  Air  Force  develops  its 
Budget  Estimate  Submission  (BES)  which  the  Office  of  the  Secretary 
of  Defense  (OSD)  and  the  Office  of  Management  and  Budget  (OMB) 
review.  The  results  of  the  OSD/OMB  review  are  formalized  by  the 
Decision  Package  Sets  (DPSs)  which  prioritize  all  DOD  programs 


according  to  the  rules  of  zero-base  budgeting.  The  final  step  in  the 


process  is  the  Major  Budget  Issues  (MBI)  meeting  between  the  SecDef, 
the  Sec  AF,  and  the  Chief  of  Staff  which  is  the  Air  Force's  final  for¬ 


mal  appeal  on  behalf  of  its  desired  programs  (AFP  172-4,  1979:33). 

Budget  Estimate  Submission 

The  Budget  Estimate  Submission  (BES)  is  the  Air  Force's 
statement  of  the  funds  needed  to  fund  all  the  programs  in  the  first 
year  of  the  POM  (AFP  172-4,  1979:31)  and  is  based  upon  the  base  line 
developed  in  the  PDM/APDM  (Clark,  1979:75).  It  is  submitted  in  a 
zero-base  budget  framework  and  consists  of  a  minimum  level  and  a 
series  of  alternative  packages  that  are  in  priority  order  and  which 
total  to  progressively  larger  budget  totals  (AFP  172-4,  1979:33).  The 
BES  is  completed  and  submitted  to  OSD/OMB  by  October  first  and 
represents  the  AF's  program  priority  structure  within  the  budget 
limits  established  by  OSD. 

OSD/OMB  Review  Process 

After  the  BES  is  submitted  to  OSD/OMB,  it  is  divided  up 
into  specific  areas  for  review  by  OSD/OMB  comptroller  staffs.  The 
ultimate  purpose  of  this  review  process  is  to  develop  the  Decision 
Package  Set  (DPS)  which  will  become  the  final  guidance  to  the  Presi¬ 
dent  when  he  is  developing  his  budget  for  Congress.  Each  review 
group  holds  a  hearing  during  which  the  Air  Force  can  defend  its 
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particular  position  on  the  programs  in  question.  At  the  end  of  the 
review  process  the  OSD/OMB  staff  members  are  in  a  position  to 
make  the  DPS  decisions  that  could  change  the  Air  Force  budget 
(Clark,  1979:82-84).  Lf  the  Air  Force  does  not  accept  the  resource 
levels  in  the  DPS,  it  can  appeal  the  decision.  After  all  the  appeals 
are  analyzed,  the  SecDef  makes  the  final  decisions  and  sends  the 
DOD's  budget  to  the  President. 

Development  of  the  President's  Budget 

The  final  step  in  the  PPBS  cycle  is  when  the  President 
receives  the  DOD's  DPSs  and  integrates  them  with  all  the  other  pro¬ 
grams  in  the  federal  government.  The  result  is  the  presentation  of 
the  unified  federal  budget  in  January  to  the  Congress  for  debate. 
After  ten  months  of  debate,  the  budget  is  finally  enacted  by  Congress 
in  October  (Clark,  1979:85-87). 


Vanguard 

Vanguard  is  a  new  planning  process  initiated  by  the  Air  Force 
Systems  Command  (AFSC)  which  will  clearly  show  the  contribution 
and  interrelationship  of  each  exploratory  development,  advanced 
development,  engineering  development,  and  acquisition  project  in 
AFSC.  Decision  makers,  using  the  information  provided  by  the  Van¬ 
guard  process,  will  be  able  to  identify  the  need  for  modified  emphasis 
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on  programs,  changes  in  program  schedules,  or  new  technology- 


developments.  The  results  from  Vanguard  will  be  a  prioritized  list 
of  efforts  which  will  be  used  as  an  input  to  the  AFSC  POM  and  BES 
exercises  (DCS/Plans  and  Programs,  1979:2-28) . 

ASP  POM  Process 

One  of  the  most  important  planning  activities  conducted  at 
ASD  during  the  fiscal  year  is  the  POM  prioritization  exercise  because 
the  results  represent  the  ASD  Commander's  views  on  the  programs 
that  comprise  his  future  business  base  (ASD/AV,  1980:15).  Following 
the  theory  of  zero-base  budgeting,  this  prioritized  list  of  programs 
is  the  view  of  the  person  who  is  held  responsible  for  those  programs. 
The  list  represents  the  ASD  Commander's  preference  for  which  pro¬ 
grams  should  be  added  if  additional  funds  become  available  and  which 
should  be  dropped  if  funds  are  cut. 

The  POM  process  is  initiated  when  the  ASD  Comptroller 
receives  POM  guidance  from  AFSC  Headquarters  in  mid-November. 
This  guidance  provides  instructions  on  how  the  decision  packages 
should  be  prepared  and  on  what  dollar  figure  should  be  used  as  the 
approved  funding  level.  The  program  managers  complete  all  the 
decision  packages  for  their  programs  and  submit  the  material  back 
to  the  Comptroller's  office.  The  Comptroller's  office  sends  a  copy 
of  all  ASD  decision  packages  to  ASD/XR.  Mission  Area  Review 
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panels  are  established  and  rank  all  the  decision  packages  within  their 
assigned  mission  area.  The  proposed  rankings  within  each  mission 
area  are  reviewed  by  the  Program  Review  Group  and  finally,  by  the 
Decision  Review  Group  which  is  chaired  by  the  ASD  Commander. 
After  the  Decision  Review  Group's  review,  ASD's  input  to  the  POM  is 
sent  to  AFSC  Headquarters  and  represents  the  ASD  Commander’s 
priority  of  programs  within  each  mission  area  (ASD/AV,  1980: 15) , 

At  least  four  decision  packages,  minimum,  decremented, 
approved,  and  enhanced,  are  required  for  each  program.  The 
minimum  level  is  defined  as  the  funding  level  below  which  it  is  not 
productive  to  continue  the  program  (approximately  70  per  cent  of 
the  approved  level).  The  decremented  level  is  the  funding  level 
between  the  minimum  level  and  the  approved  level  (approximately  90 
per  cent  of  the  approved  level)  and  may  not  permit  the  accomplish- 
f  ment  of  all  the  program  objectives.  The  approved  level  is  the  AF 

approved  funding  level  in  the  FYDP.  Finally,  the  enhanced  level  is 
an  increased  funding  level  that  is  required  in  order  to  provide  an 
earlier  initial  operation  capability  date,  lower  life  cycle  costs,  or  an 
increased  capability.  A  fifth  decision  package  could  be  submitted  by 
the  program  manager  and  represents  a  special  collection  of  projects 
which  was  not  covered  in  the  previous  four  funding  levels  (ASD 
Pamphlet  800-19,  1979:50). 
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Summary 

The  discussion  about  ZBB  provides  a  theoretical  base  line 
for  analyzing  and  discussing  problems  encountered  by  the  DWMS 
panel  when  attempting  to  evaluate  and  rank  decision  packages  for  the 
1982-1986  POM.  The  literature  mentions  five  steps  to  the  ZBB  pro¬ 
cess,  the  information  that  should  be  available  in  order  to  evaluate 
each  decision  package,  and  the  process  recommended  in  order  to 
institute  major  changes  in  an  organization.  The  results  of  interviews 
with  the  DWMS  panel  members  are  presented  in  Chapter  4  and  parti¬ 
cular  responses  compared  to  the  ZBB  base  line  in  order  to  analyze 
discrepancies  and  to  suggest  changes. 

The  PPBS  used  by  the  Department  of  Defense  was  also  dis¬ 
cussed  in  order  to  determine  what  documents  are  available  for  use  by 
the  Mission  Area  Review  panels  when  reviewing  decision  packages 
for  the  POM.  The  BES,  the  Air  Force  Planning  Guide,  and  the  P- 
series  documents  would  be  appropriate  to  review  in  order  to  provide 
additional  input  information  for  the  program  evaluation  process.  The 
major  problem  associated  with  the  DWMS  mission  area  is  that  its 
programs  support  several  of  the  other  mission  areas  and  cannot  be 
related  directly  to  the  objectives  of  those  mission  areas.  Chapter  3 
contains  a  discussion  of  the  factors  involved  when  evaluating  and 
selecting  R8i  D  proj ects  . 
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Chapter  3 

R&D  PROJECT  EVALUATION  AND  PROJECT  SELECTION/ 
RESOURCE  ALLOCATION  PROCESS 

The  R&D  project  selection  and  resource  allocation  decision 
is  a  very  complex  process  for  any  organization.  The  purpose  r  (  this 
chapter  is  to  discuss  the  various  factors  which  should  be  considered 
when  evaluating  an  R&D  project  and  various  methods  of  using  those 
factors  to  select  the  most  beneficial  combination  of  projects  for  the 
organization.  Paolini  and  Glaser  state  that  the  choice  of  the  appro¬ 
priate  project  selection  technique  depends  upon  the  nature  and  size 
of  the  research  organization,  its  objectives  and  resources,  and  the 
type  and  quality  of  data  available  (Paolini  and  Glaser,  1977:26).  The 
most  important  factor  in  considering  and  selecting  any  project  evalu¬ 
ation  and  resource  allocation  model  is  the  value  of  that  model  to  the 
organization.  If  the  benefits  received  from  the  model  are  not  greater 
than  the  costs  or  if  the  model  is  too  complex  or  requires  an  inordin¬ 
ate  amount  of  time  to  collect  the  input  data,  the  model  is  worthless 
to  the  organization,  Bemelmans  addresses  a  major  problem  in 
model  development  when  he  says  that  "there  is  no  sense  in  using 
models,  perfectly  describing  complex  research  situations,  but  at  the 
same  time  so  complex  that  an  application  would  hardly  be  possible 
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[Bemelmans,  1979:38]." 


A  major  disillusionment  many  managers  have  for  highly 
quantitative  models  is  that  these  models  suggest  a  precision  that  is 
not  there.  Most  of  the  input  data  are  based  on  subjective  evaluation 
by  either  an  individual  or  a  panel  of  experts.  The  accuracy  of  calcu¬ 
lations  is  limited  by  the  validity  of  the  figures  used  in  the  computa¬ 
tions  (Roman,  1988: 2  121 .  Paolini  and  Glaser  state  that  management 
may  be  mislead  by  quantified  models: 

Many  managers  believe  that  mathematical  models  and 
numerical  formulas  may  tend  to  mislead  by  implying  that 
quantifiable  information  is  available  that  is  more  reliable 
and  complete  than  is  actually  the  case.  The  finite  solutions 
obtained  suggest  a  precision  that  doesn't  exist  [Paolini  and 
Glaser , 1977:2b] . 

Roman  says  that  managers  should  not  rely  only  on  quantifiable  infor¬ 
mation: 


.  .  .  though  the  use  of  numerical  evaluation  gives  an 
aura  of  mathematical  accuracy  to  decisions  which  often  makes 
them  easier  to  justify  to  those  who  may  review  or  question 
them,  exclusive  reliance  on  quantification  can  convey  a  mis¬ 
leading  indication  of  both  the  relative  and  the  absolute  value 
of  the  overall  project  [Roman,  1968:212]. 

The  following  reasons  are  given  by  Roman  as  why  quantification  gives 

management  only  a  general  idea  regarding  project  selection: 

1)  the  quantification  of  data  cannot  guarantee  the  accuracy 
and  relevance  of  that  data,  and 

2)  the  quantification  of  data  cannot  be  applied  to  the  non- 
quantifiable  elements  which  must  be  considered  (Roman,  1968:212). 
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In  evaluating  and  selecting  an  appropriate  R&D  project  portfolio  both 
quantitative  and  qualitative  factors  have  to  be  considered  in  order  to 
select  the  best  possible  portfolio  for  the  organization.  If  the  model 
captures  enough  of  the  problem,  it  can  be  a  useful  input  into  the 
allocation  process. 

The  complex  environment  in  which  the  R&D  process  is  in¬ 
corporated  is  depicted  in  Figure  5.  Chiu  and  Gear  state  that  the 
selection  and  allocation  decision  involves: 

1)  a  variety  of  factors,  some  technical,  but  others  organ¬ 
izational,  behavioral,  and  economic: 

2)  multiple  and  conflicting  objectives  and  priorities  at 
various  levels  in  the  organization; 

3)  varying  degrees  of  subjectivity  in  predicting  outcomes  of 
actions  and  estimating  related  probabilities; 

4)  complex  sequential  interactions  between  projects  and 
with  the  "outside  world"  (Chiu  and  Gear,  1979:2). 
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A  study  conducted  by  Keeler  determined  that  management 


felt  that  an  evaluation/selection  model  could  contribute  to  R&D 
planning.  Management  "...  felt  it  could  be  helpful  in  achieving  a 
consensus  by  functioning  as  a  communication  device  and  by  stimu¬ 
lating  and  facilitating  rational  analysis  on  the  part  of  those  involved 
in  R&D  planning  [Keefer,  1978:14]."  Many  organizations  have  become 
disillusioned  and  turned  away  from  quantitative  models  because  these 
models  neglect  or  fail  to  evaluate  many  important  qualitative  factors. 
Baker  speculates  that  the  future  importance  for  models  will  be  away 
from  "decision  models"  and  towards  "decision  information  systems" 
(Baker,  1974:169).  Quantitative  models  will  not  be  used  to  provide 
the  final  decision  but  will  provide  additional  input  information  for  the 
decision  maker  to  evaluate  along  with  non-quantifiable  variables. 

R&D  Project  Portfolio  Concept 

In  order  to  maximize  the  benefits  to  the  organization,  the 
R&D  project  portfolio  must  be  considered  as  a  whole  because  the 
ultimate  concern  is  not  the  individual  project  performance  but  the 
total  benefit  provided  by  the  portfolio  to  the  organization.  The  pro¬ 
jects  cannot  be  evaluated  as  independent  entities  because  they  compete 
for  resources  especially  financial  and  are  therefore  interdependent. 
Twiss  makes  an  important  point  when  he  states  that: 

The  aim  of  maximizing  the  contribution  from  the  whole 
R&D  portfolio  may,  therefore,  occasionally  result  in  the 


rejection  of  a  project  which  might  otherwise  appear  attractive 
because  its  high  consumption  of  a  particular  resource  leads 
to  the  starvation  of  other  projects  also  requiring  that  same 
resource  [Twis  s ,  1974:37] . 

Seiler  discusses  another  possibility  when  a  single  large  project  may 
be  dropped  in  favor  of  several  smaller  projects  which  have  lower 
individual  benefits  but  which  provide  a  higher  total  benefit  (Seiler, 
1965:168).  In  order  to  efficiently  use  R&D  resources,  all  projects 
should  be  interdependently  evaluated  to  determine  the  best  combina¬ 
tion  or  portfolio  of  projects,  rather  than  evaluating  each  project  as 
a  mutually  exclusive  entity. 

Project  Evaluation  Criteria 

Before  a  project  can  be  evaluated,  an  appropriate  set  of 
criteria  has  to  be  established.  Roman  says  that  the  criteria  for 
selecting  projects  vary  in  degree  of  importance  according  to  the 
nature  and  the  need  of  the  organization  performing  the  evaluation 
(Roman,  1968:207) .  Beattice  and  Reader  say  that  the  project  evalu¬ 
ation  criteria  must  be  based  upon  the  particular  organization's  objec¬ 
tives  and  limitations  (Beattice  and  Reader ,  197  1:50) .  If  the  evaluation 
criteria  are  not  related  to  the  goals  and  objectives  of  the  organization, 
then  the  wrong  portfolio  of  projects  could  be  selected  and  resources 
will  be  wasted  on  projects  that  do  not  satisfy  deficiencies  of  the 
organization. 
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Many  authors  have  discussed  the  type  of  factors  that  need  to 
be  considered  when  evaluating  R&D  projects.  Twiss  lists  five  cri¬ 
teria:  corporate  objectives,  strategy,  policies,  and  values  criteria; 
marketing  criteria;  R&D  criteria;  financial  criteria;  and  production 
criteria  (T wis s ,  1974: 122) .  Blake  says  that  when  evaluating  any  R&D 
portfolio  the  following  multiple,  conflicting  criteria  occur: 

1)  evaluation  of  proposed  projects  with  respect  to  cost, 
feasibility,  need,  benefit,  and  motivation  of  people; 

2)  availability  of  personnel; 

3)  project  risk; 

4)  management's  resistance  to  risk  and  change; 

5)  technological  forecasting--prediction  of  useful  end  result 
or  advances  in  the  state  of  the  art; 

6)  biases  in  R&D  estimates; 

7)  future  costs;  and 

8)  differences  in  perceptions  between  top  management  and 
R&D  management  (Blake,  1978:32-65). 

Roman  lists  five  main  criteria:  need  satisfaction;  urgency  (immedi¬ 
acy  of  need/opportunity);  predictability  (degree  of  certainty);  per¬ 
ceived  future  effect  on  savings  in  material,  cost,  or  labor;  and 
likelihood  of  technical  success  (Roman,  1968: 166) .  Roman  also  says, 
that  for  military  projects ,  mission  accomplishment  is  of  overriding 
concern  and  that  military  projects  should  be  evaluated  with  respect  to 
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need,  schedule,  and  anticipated  performance  (Roman,  1968:207). 

R&D  projects  can  interact  in  several  different  ways  which  is 
why  it  is  important  to  evaluate  and  select  a  project  portfolio  rather 
than  selecting  projects  just  on  their  independent  merit.  Baker  and 
Freeland  in  their  review  of  project  selection  techniques  state  that 
one  of  the  most  important  limitations  of  existing  approaches  is  the 
"inadequate  treatment  of  project  interrelationships  with  respect  to 
both  value  contribution  and  resource  utilization  [Baker  and  Freeland, 
1975:1165]."  Aaker  and  Tyebjee  have  suggested  a  technique  which 
considers  three  types  of  interdependencies  among  projects.  Their 
model  is  formulated  to  include  adjustments  to  both  the  objective 
function  and  to  the  resource  constraints  (Aaker  and  Tyebjee,  1978:30- 
36). 

The  first  type  of  interdependency  is  due  to  overlap  in  project 
resource  utilization;  projects  utilize  common  equipment,  personnel 
efforts,  facilities,  etc.  The  budget  for  such  sets  of  projects  would 
be  less  than  if  those  projects  were  pursued  individually.  Second, 
two  projects  may  be  technically  interdependent  when  the  success  or 
failure  of  one  project  significantly  enhances  or  retards  the  progress 
of  other  projects.  Finally,  if  the  value  contributions  or  payoffs  for 
two  projects  are  nonadditive,  then  those  two  projects  are  considered 
effect  interdependent.  Projects  may  have  a  synergistic  effect  where 
the  benefits  to  the  organization  may  be  greatly  enhanced  if  both  are 
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selected.  Just  as  possible  is  that  projects  may  have  a  cannibalistic 


effect  where  two  projects  are  developing  systems  which  compete 
with  each  other  and  the  payoff  if  both  are  selected  is  much  less  than 
their  summed  payoff  to  the  organization  (Aaker  and  Tyebjee,  1978:30). 

Several  major  criteria  are  common  to  all  of  the  recommended 
evaluation  criteria  in  the  literature.  All  authors  mentioned,  in  some 
form,  the  importance  of  considering  long  range  organizational  goals, 
risk,  need,  cost,  technical  performance,  and  proj ect  time  phas  ing 
when  evaluating  projects. 

Long  Range  Organizational  Goals 

The  most  important  factor  to  be  considered  by  management 
is  how  the  R&D  project  relates  to  the  long-range  goals  and  objectives 
of  the  organization.  R&D  project  objectives  must  be  defined  with 
respect  to  the  goals  and  objectives  of  the  organization  so  that  the 
organization  uses  its  limited  resources  in  the  most  efficient  manner 
possible  in  achieving  those  goals  and  objectives.  Gee  says  that  it  is 
extremely  important  that  the  project  goals  are  consistent  with  the 
organizational  goals  because  the  R&D  projects  have  to  be  evaluated 
with  respect  to  their  potential  contributions  to  those  organizational 
goals  (Gee,  1971:42).  Twiss  discusses  the  point  that  one  of  the  most 
important  qualitative  criteria  is,  how  does  the  project  relate  to  the 
organization's  objectives,  strategy,  policies,  and  values?  He  states 
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that  it  is  vitally  important  that  strategic  considerations  be  reflected 


explicitly  in  the  project  selection  procedure  (Twiss,  1974:122). 

Whaley  and  Williams  state  that  "without  a  clearly  stated  project 
objective,  it  will  be  difficult  to  determine  whether  the  technical  effort 
has  met  that  objective  [Whaley  and  Williams,  1971:27]."  If  the  project 
selection  proces s  is  to  be  decentralized,  then  Villers  claims  that 
"the  long-range  objectives  of  the  company  should  be  known  to  all 
those  who  make  decisions.  Otherwise,  the  research  activity  to  a 
great  extent,  operates  in  a  vacuum  [Villers ,  1964:20] .  "  DeGreene 
discusses  the  problems  in  establishing  the  long  range  goals  and  ob¬ 
jectives  of  an  organization. 

Long  range  goals  and  objectives  must  be  defined  within 
the  context  in  which  the  organization  or  system  will  exist. 

At  a  time  of  explosive,  almost  chaotic,  change  this  is  probably 
impossible  for  a  rigid  system.  The  job,  then,  is  to  build  as 
much  adaptability  and  flexibility  into  the  system  structure, 
function,  and  behavior  as  possible.  At  the  same  time,  the 
job  becomes  even  more  complicated  by  virtue  of:  (1)  demands 
for  short-range  benefits;  and  (2)  the  fact  that  industrial 
managers  and  political  decision  makers  tend  to  perceive  pro¬ 
blems  and  goals  in  relation  to  their  term  in  office;  they  are 
rewarded  by  accomplishing  things  today,  not  by  having  set  the 
stage  for  a  better  organization  or  world  tomorrow.  There  is 
thus  the  very  real  danger  of  near-raage  over  exploitation  of 
resources,  resulting  in  greatly  increased  long-range  costs  to 
pay  for  short-range  benefits  [DeGreene,  1973:173], 

An  R&iD  project  has  to  be  continually  evaluated  in  terms  of 
the  current  long  range  goals  of  the  organization.  Since  an  R&rD  pro¬ 
ject  may  last  many  years  and  budget  decisions  are  usually  made  once 
a  year,  it  is  important  to  evaluate  each  project  with  respect  to  the 


r 


current  and  future  organizational  goals  not  the  goals  when  the  project 
was  initiated.  Due  to  technological  break  throughs  or  policy  changes, 
organizational  goals  can  change  over  time. 

Risk 

R&cD  projects  involve  greater  risk  than  other  types  of  pro¬ 
jects  that  an  organization  can  undertake.  Seiler  discusses  three 
types  of  risks  associated  with  R&D  projects.  The  first  is  the  risk 
that  the  selected  project  may  net  be  successful;  that  is,  what  is  the 
probability  of  technical  success  of  the  project?  The  probability  of 
technical  success  is  a  function  of  the  state  of  the  art  in  all  fields 
relevant  to  the  project  being  considered.  The  second  type  of  risk  is 
the  probability  that  the  project  is  successful  but  the  need  no  longer 
exists  once  the  project  is  completed.  Since  R&D  projects  take  many 
years  to  complete,  by  the  time  the  project  is  completed  the  original 
need  may  have  changed.  The  relevant  question  here  is,  what  is  the 
probability  of  technical  obsolescence  during  development?  The  last 
type  of  risk  is  the  probability  that  a  rejected  project  may  have  been 
successful  if  it  had  been  selected  {Seiler,  1965:1291. 

Another  type  of  risk  occurs  due  to  the  subjective  nature  of 
the  input  parameters  that  are  used  to  evaluate  each  project  and  tends 
to  produce  biases  in  the  R&D  estimates.  Blake  states  that  biases 


enter  into  the  estimation  of  the  following  parameters:  probability  of 


technical  success,  proposed  development  time,  estimated  develop¬ 
ment  costs,  and  perceived  need  for  the  project.  He  says  that  the 
person  doing  the  estimate  usually  has  an  incentive  to  estimate  low  in 
order  to  sell  the  project.  Estimates  of  time  and  costs  will  probably 
never  be  accurate  so  decision  makers  should  always  keep  in  mind  how 
poor  their  data  is  (Blake,  1978:57). 

Need 

An  organization  develops  a  long  range  strategic  plan  to  pro¬ 
vide  direction  to  that  organization  for  several  years  into  the  future. 
Along  with  developing  that  plan,  planners  project  current  capabilities 
into  the  future  in  order  to  determine  if  those  current  capabilities  will 
allow  the  organization  to  achieve  its  strategic  plan.  If  current  capa¬ 
bilities  fall  short  of  the  required  capabilities,  then  a  deficiency  exists. 
Ranking  projects  and  allocating  resources  according  to  projected 
deficiencies  would  not  be  the  most  efficient  allocation  of  an  organi¬ 
zation's  scarce  resources.  In  order  to  determine  an  organization's 
need,  the  relative  importance  of  satisfying  each  deficiency  has  to  be 
considered.  It  would  not  be  in  the  best  interest  of  the  organization  to 
allocate  resources  to  an  area  which  has  a  large  deficiency  but  is  con¬ 
sidered  to  be  of  little  importance  to  that  organization.  Need  is  deter¬ 
mined  by  multiplying  the  deficiency  in  a  particular  area  by  the  impor¬ 
tance  to  the  organization  of  satisfying  that  need  (Belcher,  1980). 


Roman  states  that  need  satisfaction  is  one  of  the  most  important  cri¬ 
teria  in  evaluating  R&D  projects  and  is  of  overriding  importance  to  a 
military  R&D  selection  process  (Roman,  1968:207). 

Costs 

When  considering  the  worth  of  any  R&D  project,  the  decision 
maker  wants  to  maximize  the  future  benefits  per  dollar  spent  on  RhD. 
Because  many  R&D  project  estimates  are  highly  variable,  it  is  diffi¬ 
cult  to  obtain  an  accurate  estimation  for  costs.  Since  the  R&D  envi¬ 
ronment  is  so  fluid,  Blake  states  that  in  order  to  obtain  an  accurate 
reason  for  cost  increases  any  changes  in  costs  due  to  changes  in  the 
scope  of  the  work  must  be  separated  from  cost  changes  due  to  poor 
management,  inaccurate  cost  estimates,  or  changes  in  material  costs 
and  direct  labor  rates  (Blake ,  1978:69) .  He  also  addresses  the  issues 
of  sunk  costs  and  project  termination.  Economic  theory  says  that 
decisions  should  be  made  between  alternatives  that  can  affect  the 
future  and  money  already  spent  should  not  be  considered.  Consider¬ 
ing  sunk  costs  is  not  valid  for  deciding  whether  to  spend  more  money 
on  a  project  or  to  terminate  that  project.  Bobis  etal.  state  that 
expenditures  to  date  should  not  be  considered;  therefore,  each  pro¬ 
ject  must  be  reconsidered  at  budget  time  and  that  money  already 
spent  is  gone.  Anticipated  returns  are  based  only  on  the  research 
expenditures  that  are  still  required  and  the  potential  effect  of  the 
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project  on  future  benefits  to  the  organization  (Bob is  e t  a  1 .  ,  197  1:36). 

Time  Phase 

Most  authors  discussed  the  importance  of  considering  the 
time  phase  of  the  R&D  projects  when  determining  the  appropriate 
R&D  portfolio.  Seiler  describes  timing  in  terms  of,  will  the  funds  be 
available  when  needed;  and  will  the  research  be  completed  and  the 
product  be  developed  in  order  to  satisfy  the  organization’s  need  at 
the  appropriate  time  (Seiler,  1965:138)?  Roman  says  that  timing  is 
critical  in  determining  the  urgency  of  the  project.  If  the  need  is 
immediate,  then  the  project  will  receive  a  high  priority  and  a  large 
portion  of  the  available  funds  (Roman,  1968:  166) .  In  the  military 
environment  timing  is  very  important.  Will  the  particular  concept 
become  an  operational  system  in  time  to  meet  a  perceived  threat,  or 
will  a  large  amount  of  money  be  spent  on  a  system  which  will  be 
obsolete  before  it  becomes  operational? 

Technical  Performance 

Of  course  when  evaluating  an  R&D  project,  the  technical 
specifications  and  changes  to  those  specifications  have  to  be  con¬ 
sidered.  Will  the  end  product  be  able  to  perform  as  stated  in  the 
project  documentation?  Gee  states  that  the  project  should  be  evalu¬ 
ated  with  respect  to  the  technical  soundness  of  the  proposal  and 


whether  it  is  within  the  current  state  of  the  art  or  requires  a  tech¬ 
nological  break  through  (Gee,  1971:42),  Roman  says  that  for  a  mili¬ 
tary  project  the  anticipated  performance  is  of  major  importance  in 
determining  whether  the  final  product  will  be  able  to  accomplish  its 
mission  (Roman,  1968:208).  Within  the  DOD,  at  each  milestone  a 
major  system  has  to  be  evaluated  as  to  whether  or  not  it  is  meeting 
the  required  performance  specifications  (DOD  Directive  5000.  1, 
1978:3).  If  a  project  is  not  meeting  the  performance  specifications 
and  a  more  beneficial  project  is  proposed,  the  original  project  should 
be  terminated  in  favor  of  the  new  one. 

Associated  with  the  technical  performance  of  a  project  is  the 
question  of  what  future  developments  might  occur  from  the  current 
research.  Twiss  identifies  two  types  of  future  and  unplanned  benefits 
that  may  result  from  current  research  projects.  Synergy  between 
projects  occurs  when  the  research  conducted  on  one  project  helps  to 
solve  problems  on  a  completely  different  project.  Lessons  learned 
reports  or  informal  meetings  can  pass  along  information  that  is 
beneficial  to  all  researchers.  Another  result  is  that  of  project  spin¬ 
offs.  Research  on  one  project  may  result  in  unforeseen  benefits  or 
applications  to  an  entirely  different  field  of  technology  (Twiss,  1978; 
138). 
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Military  Project  Selection 

Although  many  of  the  same  type  of  criteria  are  important  in 
evaluating  both  commercial  and  military  R&D  programs,  there  are 
substantial  differences  because  military  project  ideas  are  developed 
in  response  to  a  perceived  need  rather  than  to  a  profit  motive. 

Roman  states  that  the  most  compelling  force  in  project  selection  is 
the  identification  and  justification  of  the  need  relative  to  the  cost 
and  the  potential  value  of  the  project.  He  goes  on  to  discuss  four 
elements  which  must  be  weighed  in  the  decision  making  process: 

11  technologically  feasibility, 

2)  expected  military  value  of  the  system, 

3)  expected  cost,  and 

4)  value  and  cost  compared  with  those  of  existing  or  other 
possible  systems  (Roman,  1968:2 13) . 

Fundamental  in  any  weapon  system  decision  are:  the  threat 
of  a  potential  aggressor,  the  cost  of  the  system,  the  existing  state  of 
the  art,  and  timing.  The  state  of  the  art  not  only  determines  the  pre¬ 
sent  ability  to  deal  with  possible  threats  but  also  the  quality  of  the 
possible  R&D.  At  any  point  in  time  and  under  a  given  circumstance, 
any  one  of  the  fundamental  elements  could  dominate  the  selection 
process  (Roman,  1968: 2  1 3 - 220 ) .  Blake  notes  that  military  projects 
are  characteristically  more  technological  innovative  and  therefore 
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greater  risks  are  involved  compared  to  civilian  projects  (Blake, 

1978:53). 


R&D  Project  Selection  Techniques 

Once  all  the  R&cD  projects  available  to  the  organization  have 
been  evaluated  and  the  perceived  benefits  determined,  the  next  pro¬ 
blem  for  the  decision  maker  is  how  to  select  the  best  portfolio  of 
projects  given  the  resource  constraints.  Many  models  have  been 
discussed  that  vary  from  a  simple  evaluation  by  one  decision  maker 
to  complex  mathematical  programming  models.  As  more  people 
study  the  R&D  selection  proces  s  and  as  more  accurate  input  data 
becomes  available,  more  decision  makers  have  turned  away  from 
qualitative  models  toward  quantitative  models.  Another  factor  driving 
decision  makers  toward  quantitative  models  is  that  as  resources  be¬ 
come  more  limited  all  organizations  need  to  obtain  the  maximum 
possible  benefit  for  each  dollar  spent  on  R&D. 

Scoring  Models 

One  of  the  o  iginal  methods  for  evaluating  R&D  projects  was 
with  a  scoring  model.  Baker  and  Freeland  state  that  in  a  scoring 
model  it  is  assumed  that  a  relatively  small  number  of  decision  criter¬ 
ia  can  be  defined  to  effectively  evaluate  the  worth  of  each  project. 

The  criteria  are  related  to  specific  characteristics  of  the  project, 
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and  a  group  of  experts  determines  the  relative  worth  of  each  project. 
The  vectors  of  relative  worths  from  each  expert  are  either  summed 
or  multiplied  together  to  form  a  total  score  for  each  project.  This 
total  score  represents  the  group’s  opinion  of  the  relative  worth  of 
that  project.  Projects  are  ranked  by  their  total  score  and  included 
in  the  portfolio  according  to  the  priority  ranking  until  some  constraint 
manpower,  facilities,  or  budgetary,  is  exceeded  (Baker  and  Freeland 
1975:1168). 

Bemelmans  lists  three  main  advantages  of  scoring  models. 
First  is  the  consideration  of  both  qualitative  and  quantitative  data 
at  the  same  time  because  both  types  of  data  are  converted  to  a  single 
number  by  the  use  of  an  interval  scaled  evaluation  criteria.  Secondly 
scoring  models  are  simple  and  easy  to  apply.  Finally,  it  forces 
everyone  to  consider  the  same  relevant  aspects  of  the  project 
(Bemelmans,  1979:38). 

Three  disadvantages  are  mentioned  by  Bemelmans.  First, 
the  scores  suggest  a  precision  that  is  not  really  there.  The  evalu¬ 
ation  process  is  highly  subjective,  and  therefore  the  results  are 
highly  sensitive  to  the  opinions  of  the  evaluators.  Secondly,  con¬ 
tending  projects  can  not  be  evaluated  in  a  detailed  and  refined  way 
(Bemelmans  ,  1979: 39) .  Harrington  mentions  that  since  the  criteria 
ratings  are  combined  into  one  total,  dimensionless  score,  little 
analysis  can  be  performed  as  to  the  degree  that  the  resulting  portfolio 


meets  various  organizational  criteria  (Harr ington ,  1979:  lb) .  The  net 
result  could  be  an  undesired  portfolio.  Finally,  Bemelmans  men¬ 
tions  the  lack  of  model  structure  between  alt  evaluation  aspects. 
Interrelationships  and  interdependencies  between  projects  and  criteria 
have  a  tendency  to  be  overlooked  or  underestimated  (Bemelmans, 
1979:39).  Another  disadvantage  is  that  once  the  budget  constraint 
becomes  binding  the  portfolio  is  selected.  Scoring  models  neglect 
the  possibility  that  by  exchanging  two  rejected  projects  for  one 
selected  project,  a  better  portfolio  could  result  if  the  summed  total 
benefits  of  the  rejected  projects  are  greater  than  the  benefit  of  the 
selected  project  even  though  their  individual  benefits  are  lower. 

Economic  Models 

Many  economic  models  exist  in  the  literature  and  have  been 
tried  with  limited  success.  In  one  model,  an  index  of  relative  worth, 
based  upon  economic  criteria,  is  calculated  for  each  project  and  then 
the  projects  are  rank  ordered  according  to  this  index.  In  some  cases, 
a  minimum  index  value  is  established  below  which  the  project  is 
rejected.  All  projects  above  the  minimum  index  value  are  selected 
based  upon  their  ranking  until  the  budget  constraint  is  satisfied. 

Seiler  discusses  a  profitability  index  which  is  a  ratio  of  expected 
profit  to  expected  total  costs  for  the  project.  The  probability  of 
technical  success  is  multiplied  by-  the  probability  of  commercial 
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success,  annual  sales  in  dollars,  and  the  expected  market  life  and 
represents  the  expected  profit  of  the  project.  The  expected  total 
cost  is  the  sum  of  the  R&D  costs,  the  development  costs,  and  the 
marketing  costs  (Seiler,  1965: 16) . 

Other  economic  models  are  based  upon  the  capital  budgeting 
concept.  Paolini  and  Glaser  discuss  models  based  upon  either  the 
internal  rate  of  return  (IRR)  or  the  net  present  value  (NPV)  concept. 
The  portfolio  is  developed  by  ranking  all  projects  by  either  their  IRR 
or  NPV  and  including  all  projects  up  to  the  budget  constraint. 

Although  these  concepts  seem  initially  appealing  especially  to  finan¬ 
cially  oriented  managers,  problems  exist  in  estimating  the  appro¬ 
priate  interest  rate  and  the  expected  benefits  over  time.  The  capital 
budgeting  methods  also  neglect  a  factor  which  estimates  the  probability 
of  technical  success  of  the  project  (Paolini  and  Glaser,  1977:27). 

Harrington  lists  three  other  disadvantages  to  economic 
models.  First,  only  the  budget  constraint  is  considered  in  deter¬ 
mining  the  project  portfolio.  Second,  the  models  disregard  noneco¬ 
nomic  objectives  of  the  enterprise  and  select  projects  based  only  upon 
economic  criteria.  Finally,  no  guidance  is  provided  when  two  pro¬ 
jects  have  the  same  index  but  have  varying  resource  requirements 
over  time  (Harrington,  1979: 19) . 

Mathematical  Programming  Models 

Mathematical  programming  models  are  the  most  recent 
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attempt  at  developing  a  decision  model  to  aid  managers  in  the  R&D 
project  selection  process.  Major  advantages  of  these  models  over 
scoring  models  and  economic  models  are  that  mathematical  program¬ 
ming  models  select  an  optimal  portfolio  of  projects  rather  than  eval¬ 
uating  each  project  individually  and  the  formulation  allows  projects 
to  be  constrainted  by  more  than  one  resource  limitation.  For  exam¬ 
ple,  constraints  due  to  manpower,  facilities,  and  budget  resources 
are  possible  with  the  mathematical  programming  models.  In  some 
cases,  the  portfolio  may  not  be  constrained  by  the  dollars  available 
but  the  organization  may  not  have  the  laboratory  facilities  to  handle 
all  the  proposed  projects. 

A  linear  programming  (LP)  formulation  of  the  project 
selection  decision  has  been  formulated  by  Bell  and  discussed  by 
Gear  et  al.  (Gear  et  al.,  1971:66).  In  Bell’ s  model,  each  project  or 
project  version  is  represented  by  a  variable  that  may  take  any  value 
between  zero  and  one.  In  the  final  solution,  a  value  of  zero  repre¬ 
sents  a  project  that  was  not  selected  while  a  value  of  one  represents 
total  selection  of  that  project.  Any  value  between  zero  and  one 
represents  the  fraction  of  that  project  which  is  selected.  Other 
features  of  Bell's  model  are  the  consideration  of  the  multiple  time 
period  nature  of  R&D  projects  and  the  possibility  of  multiple  resource 
constraints  on  the  optimal  portfolio.  Bell's  formulation  is  typical  of 
the  LP  approach  to  portfolio  selection: 
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2.  Mutually  Exclus ive  Project  Versions 
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where: 


3.  Non-negativity 


n  is  the  number  of  projects, 

is  the  number  of  versions  of  project  i. 

N  is  the  number  of  resource  categories. 

P  is  the  number  of  time  periods. 

b..  is  the  value  of  version  j  of  project  i. 

aijkp  ’'s  tbe  arriount  resource  k  required 

by  version  j  of  project  i  in  time  period  p. 
Akp  is  the  overall  availability  of  resource 
type  k  in  period  p. 


/ 


Although  LP  formulations  allow  large  problems  to  be  effi¬ 
ciently  and  quickly  solved,  this  formulation  only  makes  sense  when 
fractional  parts  of  projects  are  feasible.  In  some  cases  fractional 
projects  could  be  included  in  the  research  effort  as  partial  versions 
of  the  original  projects.  Harrington  discusses  Asher's  claim  that 
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this  approach  shows  that  some  work  should  be  accomplished  on  the 
project  in  the  current  period  rather  than  eliminate  the  project  which 
would  allow  slack  to  occur  in  the  binding  constraints.  Several  authors 
mention  the  possibility  of  rounding  the  fractional  value  to  the  nearest 
integer  value,  but  that  is  dangerous  because  it  could  result  in  either 
an  infeasible  or  feasible  but  not  optimal  solution  (Harrington,  1979:22). 
One  important  advantage  to  the  LP  approach  is  that  much  useful 
information  such  as  shadow  pricing,  right-hand  side  ranging,  and 
benefit  row  ranging  can  easily  be  obtained  through  sensitivity  and 
postoptimality  analysis  (Gear  et  al .  ,  197  1:67) . 

Since  in  some  cases  fractional  projects  do  not  make  sense  or 
management  may  not  mind  resource  slack  in  the  uncertain,  dynamic 
R&D  environment,  the  LP  method  may  not  be  the  proper  approach  to 
R&cD  project  selection.  A  0-1  integer  programming  (IP)  formulation 
may  be  the  appropriate  approach.  Paolini  and  Glaser  discuss  the 
R&D  project  selection  problem  formulated  as  a  0-  1  IP  model  where 
the  variable,  x-. ,  is  further  constrained  as  follows: 

1  if  version  j  of  project  i  is  selected, 

x..  = 
ij 

0  otherwise,  i=  l,2,...,n, 

3  ~  1,2,..., m ^ 

(Paolini  and  Glaser,  1977:27). 

Additional  constraints  can  be  formulated  to  allow  for  the  inclusion  of 
compulsory  projects ,  compulsory  projects  with  lower  bounded  project 
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versions,  alternate  compulsory  projects,  and  mutually  dependent 
projects . 

The  major  advantage  of  the  0-1  IP  approach  is  that  it  elimin¬ 
ates  the  problem  of  selecting  fractional  projects.  The  major  dis¬ 
advantage  is  that  existing  solution  algorithms  are  relatively  very  time 
consuming  when  solving  large  problems.  Another  disadvantage  is 
that  in  order  to  perform  sensitivity  analysis  the  complete  problem 
has  to  be  rerun  (Harrington,  1979:23). 

Multicriteria  Models 

One  of  the  major  disadvantages  to  either  the  LP  or  0-1  IP 
formulation  discussed  so  far  is  that  they  do  not  explicitly  consider  the 
multiple,  conflicting  nature  of  organizational  goals.  The  multiple 
goals  are  reduced  to  a  single  objective,  which  is  usually  economic, 
for  optimization.  McMillan  states  that  managers  tend  to  set  goals 
and  then  seek  that  allocation  of  resources  which  offers  the  best  pro¬ 
mise  of  achieving  those  goals.  It  is  possible  all  the  goals  that  have 
been  established  lie  outside  the  feasible  region  for  the  given  con¬ 
straints,  and  therefore,  no  goals  can  be  achieved  exactly.  Manage¬ 
ment  would  then  seek  that  solution  which  minimizes  the  deviations 
from  those  established  goals  (McMillan,  1975:588-58°*! .  If  manage¬ 
ment  can  establish  an  ordinal  ranking  for  noncommensurate  goals, 
then  a  goal  programming  (GP)  formulation  can  be  used  to  solve  the 
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R&D  project  selection  problem. 

The  concept  of  goal  programming  is  that  a  quantifiable  goal 
is  set  for  each  of  the  multiple  objectives  relevant  to  the  problem  being 
analyzed.  Next,  preemptive  priorities  are  established  for  each  goal 
in  order  to  preserve  the  ordinal  goal  ranking  established  by  manage¬ 
ment.  The  preemptive  goal  priorities  assure  that  lower  priority 
goals  will  only  be  addressed  after  higher  priority  goals  have  been 
satisfied  as  best  as  possible.  With  multiple  goals,  all  goals  usually 
cannot  be  achieved  exactly.  Goal  programming  seeks  that  solution 
which  comes  as  close  as  possible  to  satisfying  all  of  the  quantified 
goals  by  allowing  under  and  over  attainment  of  goals.  Harrington 
discusses  the  application  of  0-1  goal  programming  to  the  selection  of 
an  R&D  project  portfolio  (Harrington,  1979:1-160). 

Besides  the  advantage  of  allowing  for  multiple,  conflicting 
goals,  goal  programming  allows  sensitivity  analysis  to  be  performed 
on  the  preemptive  goal  priorities  themselves.  Management  can 
easily  change  the  goal  priorities  and  evaluate  what  effect  that  chance 
has  on  the  final  R&D  project  portfolio.  Possibly  management  is 
uncertain  about  what  the  actual  priorities  should  be  and  sensitivity 
analysis  would  allow  them  to  determine  those  goals  to  which  the  port¬ 
folio  is  the  most  sensitive.  Also,  this  analysis  could  aid  management 
in  determining  the  best  R&D  strategy  for  the  organization  to  follow. 
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Multiperiod  Project  Modeling 


Since  R&D  projects  usually  last  more  than  one  time  period, 
the  sequential,  multiperiod  nature  of  the  projects  should  be  accounted 
for  in  the  project  selection,  resource  allocation  problem.  Hess' 
model  uses  a  dynamic  programming  approach  to  allocate  resources 
between  projects  now  and  in  the  future.  A  major  disadvantage  of  the 
model  is  that  budget  constraints  in  periods  other  than  the  first  are 
not  considered.  Secondly,  the  only  resource  constraint  that  is  con¬ 
sidered  is  the  budget.  Facilities  and  manpower  are  not  considered 
(Gear  et  al.  ,  1971:72).  Harrington  discusses  a  project  tree  format 
that  accounts  for  the  multiperiod  aspect  of  R&D  projects.  The  project 
tree  represents  the  evolution  of  projects  or  project  versions  and 
resource  requirements  and  is  used  to  develop  the  constraints  for  a 
linear  or  integer  programming  selection  technique  (Harrington,  1979: 
25). 


0-1  GP  Approach  to  ZBB 

McLaughlin  et  al.  discuss  a  0-1  goal  programming  approach 
to  the  selection  of  decision  packages  during  the  zero-based  budgeting 
process  for  a  federal  agency.  During  the  ZBB  process,  at  least  two 
mutually  exclusive  levels  of  effort  (decision  packages)  are  developed 
for  each  possible  project.  Each  level  of  effort  consumes  a  different 


amount  of  the  available  resources  and  provides  a  different  level  of 
benefits  to  the  objectives  of  the  organization.  Since  all  projects  are 
competing  for  the  same  limited  resources,  the  projects  are  inter¬ 
dependent  with  respect  to  the  total  value  of  the  possible  portfolio. 
Traditional  ranking  methods  based  upon  the  use  of  expert  panels 
and  scoring  methods  have  neglected  this  interdependency.  The  ad¬ 
vantage  of  McLaughlin's  approach  is  that  it  does  allow  for  project 
interdependencies  and  selects  the  best  portfolio  of  decision  packages 
based  upon  the  preemptive  goals  established  by  the  organization 
(McLaughlin  et  al. ,  1980: 1-27) . 

Although  McLaughlin  et  al .  formulate  their  model  as  a  0-1 
goal  programming  model,  the  same  problem  can  be  reformulated  as 
a  0-1  integer  programming  problem  by  maximizing  only  one  objec¬ 
tive- -maximize  the  expected  benefits  to  the  organization  from  the 
project  portfolio.  The  0-1  IP  can  be  formulated  very  similarly  to 
Bell's  LP  formulation  except  that  is  the  number  of  decision  pack¬ 
ages  for  project  i,  bjj  is  the  value  of  decision  package  j  of  project  i, 
aijkp  '•s  t^ie  arnount  of  resource  k  required  by  decision  package  j  of 
project  i  in  time  period  p,  and  X-  can  take  only  the  values  of  zero 
(decision  package  j  if  project  i  is  not  selected)  and  one  (decision 
package  j  if  project  i  is  selected). 
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Summary 

This  chapter  discussed  many  of  the  factors  that  should  be 
considered  when  evaluating  R&D  projects  and  different  project  selec¬ 
tion  techniques  which  can  be  used  to  select  the  best  project  portfolio 
for  the  organization.  Before  these  concepts  are  used  to  develop  a 
project  evaluation  and  resource  allocation  model  for  the  DWMS 
review  panel,  a  detailed  discussion  of  the  DWMS  mission  area  and 
the  comments  of  the  DWMS  panel  members  who  were  interviewed  are 
presented  in  the  next  chapter. 
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Chapter  4 


DEFENSE  WIDE  MANAGEMENT  AND 
SUPPORT  MISSION  AREA 

The  Defense  Wide  Management  and  Support  (DWMS)  mission 
area  is  one  of  the  ten  mission  areas  defined  in  the  AF  Systems  Com¬ 
mand  Vanguard  documentation  (DCS/Plans  and  Programs,  1979:9). 

The  programs  in  this  mission  area  provide  a  support  function  for  the 
other  six  mission  areas  and  help  contribute  to  the  objectives  of  those 
mission  areas.  During  t-he  POM  ranking  process,  the  DWMS  mission 
area  programs  are  the  most  difficult  to  evaluate  and  rank  because  of 
their  interdependence  with  the  other  mission  areas.  There  is  no 
explicit  statement  tying  DWMS  programs  to  combat  mission  area 
deficiencies.  This  chapter  describes  the  type  of  programs  included 
in  the  DWMS  mission  area,  the  composition  of  the  1982-  1986  POM 
DWMS  Mission  Area  Review  panel,  and  the  results  from  a  survey 
which  investigated  members'  attitudes  towards  and  recommended 
improvements  for  the  DWMS  POM  ranking  process. 

Defense  Wide  Management  &  Support  Mission  Area 

The  DWMS  mission  area  has  two  main  objectives.  The  first 
objective  is  to  provide  support  functions  which  are  common  to  more 
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than  one  combat  mission  area.  The  second  objective  is  to  improve 
the  quality  of  that  support  and  to  reduce  operation  and  maintenance 
costs.  Programs  proposed  to  meet  these  objectives  must  be  evalu¬ 
ated  and  justified  on  the  basis  of  their  cont:  ibution  to  the  accomplish¬ 
ment  of  combat  mission  areas  goals.  These  support  programs  must 
be  evaluated  in  terms  of  the  effectiveness  and  efficiency/economy 
they  provide  to  the  combat  forces  (Weber,  1980a). 

The  type  of  programs  included  in  the  DWMS  fall  into  nine 
major  types  of  activities. 

a.  Studies  and  Analysis.  These  programs  provide  funds  for 
development  planning  studies  to  assist  in  identifying  and  selecting  the 
more  promising  approaches  to  needed  capabilities. 

b.  Acquisition  Support.  These  programs  provide  funds  for 
telecommunications,  civilian  pay,  travel,  and  other  overhead  asso¬ 
ciated  with  weapon  system  acquisition  and  command  support  roles. 

c.  Test  and  Evaluation.  This  activity  provides  for  the  oper¬ 
ation,  maintenance,  and  improvements  for  the  4950th  Test  Wing  at 
Wright-Patterson  AFB. 

d.  Safety/Protection.  These  programs  address: 

(1)  Our  capability  to  withstand  and  recover  from  overt  or 
covert  conventional  attack. 

(2.)  Protection  from  chemical/bioloeical  agents  while 
sustaining  essential  operations. 
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(3)  Improving  the  physical  security  of  mission  essential 
facilities  /weapons /alert  aircraft. 

(4)  Life  support  equipment  for  manned  aircraft  and  haz¬ 
ardous  ground  environments. 

(5)  Aircraft  escape/descent  equipment. 

(6)  Personal  survival/recovery  equipment. 

(7)  Improved  aircraft  firefighting  equipment. 

(8)  Survivability/vulnerability  testing/assessment/ 
design  of  aeronautical  and  communications  equipment  in  a  nuclear 
warfare  environment. 

(9)  Examination  of  new  concepts  of  threats  for  defensive 
consideration  as  well  as  offensive  exploitation. 

e.  Mission  Effectiveness .  These  programs  enhance  capa¬ 
bilities  through  aircraft  non-nuclear  survivability  considerations  and 
systematic  incorporation  of  evolving  technology  in  aircraft  engines, 
and  consolidation/automation  of  aircraft  test  equipment. 

f  .  New  Capabilities.  The  next  generation  trainer  aircraft. 

g.  Equipment  Improvement.  These  programs  address 
improvements  in  current  operational  systems  and  facilities  to  correct 
identified  deficiencies. 

h.  Other.  Contingency  funding  for  future  R&tD  options 
(Weber,  1980a). 

The  DWMS  mission  area  panel  has  to  contend  with  a  major 
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problem  when  evaluating  the  DWMS  program  elements  (PE)  against 
each  other  in  order  to  make  project  tradeoff  and  resource  allocation 
decisions.  Not  only  can  one  DWMS  PE  support  more  than  one  project 
in  a  specific  combat  mission  area  but  also  a  DWMS  PE  can  support 
several  projects  across  combat  mission  areas.  Some  DWMS  PE's 
contribute  directly  to  the  achievement  of  the  goals  for  the  other  mis¬ 
sion  areas  while  for  other  DWMS  PE's  the  goal  relationship  is  vague 
and  imprecise.  Suver  and  Brown  state  that  problems  can  occur  when 
management  tries  to  compare  outputs  of  non-related  decision  pack¬ 
ages  as  is  required  in  ranking  DWMS  decision  packages  (Suver  and 
Brown,  1977:8  1) . 

Another  problem  is  that  a  DWMS  project  can  contain  all 
three  of  the  project  interdependencies  as  discussed  by  Aaker  and 
Tyebjee  (Aaker  and  Tyebjee,  1978:30).  Many  of  the  program  elements 
deal  with  essentially  very  similar  systems  and  problems,  and  if 
information  was  passed  between  program  elements,  synergy  would 
increase  the  benefits  to  the  AF.  A  DWMS  project  may  be  technically 
interdependent  upon  a  laboratory's  basic  research  results.  If  the 
laboratory's  project  fails,  the  DWMS  project  should  not  be  funded. 
How  can  one  effectively  evaluate  two  completely  different  types  of 
program  elements  which  involve  completely  different  technologies 
when  the  relationship  between  organizational  goals  and  DWMS  pro¬ 
grams'  objectives  are  not  clear? 
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Categories  of  DWMS  Program  Elements 


The  DWMS  PE's  can  be  grouped  into  several  different  cate¬ 
gories  depending  upon  the  type  of  program  and  guidance  provided  by 
higher  management.  The  first  category  includes  "level  of  effort" 
projects.  These  projects  have  a  high  return  on  investment,  low  or 
negligible  technical  risk,  and  the  number  of  projects  that  are  funded 
depend  upon  the  level  of  funding  approved  for  that  program  element. 
The  next  category  of  projects  include  the  compulsory  projects . 

These  are  either  "letters  of  agreement"  projects  or  projects  for 
which  the  appropriate  decision  package  is  specified  by  higher  manage¬ 
ment.  "Letters  of  agreement"  projects  are  joint  projects  with 
another  unit  of  the  AF,  another  service,  or  any  other  agency,  and 
the  AF  has  committed  itself  to  a  certain  level  of  funding  support. 

The  third  category  consists  of  three  management  and  support  program 
elements.  These  PE's  provide  the  funds  for  ASD  telecommunications, 
ASD  overhead,  and  the  4950th  Test  Wing  overhead.  The  last  category 
of  programs  are  acquisition  and  development  programs  in  support  of 
the  combat  mission  area  objectives  and  have  varying  degrees  of 
technical  risk  and  return  on  investment.  With  such  a  wide  diversity 
of  programs,  it  is  a  difficult  process  for  the  DWMS  panel  to  evaluate 
and  rank  all  the  program  elements'  decision  packages  according  to 
their  contribution  to  AF  objectives  for  the  ASD  POM  input. 
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DWMS  Mission  Area  Review  Panel 


The  purpose  of  the  DWMS  Mission  Area  Review  panel  is  to 
evaluate  and  priority  rank  all  the  decision  packages  within  the  mission 
area  according  to  their  benefits  to  the  AF.  This  evaluation  and 
priority  ranking  process  is  the  first  step  in  the  federal  government's 
zero-base  budgeting  process.  The  benefits  to  the  AF  of  each  decision 
package  have  to  be  determined  by  the  panel  based  upon  project  infor¬ 
mation  provided  by  the  program  managers  and  AF  goals,  needs,  and 
deficiencies  provided  by  AF  PPBS  documentation. 

Because  this  mission  area  contains  such  a  diverse  set  of 
project  types  from  which  to  select  a  portfolio,  the  panel  of  experts, 
which  was  chosen  to  evaluate  the  projects  for  the  1982-1986  POM, 
consisted  of  a  representative  from  each  major  organization  which  had 
a  program  in  the  DWMS  mission  area.  The  panel  was  composed  of 
ten  people:  eight  representatives  from  each  organization,  a  chair¬ 
man,  and  a  vice-chairman.  The  chairman  and  vice-chairman  had  a 
general  overview  level  of  knowledge  about  the  programs  rather  than 
detailed  knowledge  about  any  one  program.  The  eight  representatives 
had  detailed  knowledge  about  their  organization's  programs  but  little 
to  no  knowledge  about  the  other  programs  in  the  mission  area.  Suver 
and  Brown  make  the  point  that  the  managers  whose  packages  are 
being  ranked  should  be  members  of  the  evaluation  panel  so  that  if 
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their  project  is  cut  from  the  budget  they  will  fully  understand  the 
justification  for  that  cut  (Suver  and  Brown,  1977:78). 

Previous  experience  in  the  POM  ranking  process  ranged  from 
none  to  three  years.  Three  members  had  never  heard  of  zero-based 
budgeting;  four  members  had  heard  of  it  and  had  limited  knowledge 
about  ZBB  concepts;  and  three  members  had  a  good  working  know¬ 
ledge  of  the  ZBB  philosophy.  However,  none  of  the  members  had  had 
any  formal  training  in  ZBB  procedures  prior  to  the  formation  of  the 
panel.  Cheek  recommends  that  top  management  provide  detailed 
guidelines  and  training  to  the  people  who  will  be  performing  the  ZBB 
process  before  the  process  begins  (Cheek,  1978:24).  Without  the 
training  and  guidelines,  there  is  no  guarantee  that  the  decision  pack¬ 
ages  recommended  to  top  management  will  provide  the  best  portfolio 
of  projects  in  order  to  meet  the  organization's  objectives. 

An  important  factor  in  the  success  of  the  DWMS  panel  was 
the  experience  of  the  panel  members.  All  members  had  many  years 
of  experience  in  the  acquisition  process  at  ASD  and  were  key  members 
in  the  organizations  which  they  represented.  Intuition  by  the  panel 
members  was  an  important  factor  in  the  decision  process. 

DWMS  Interview 

Since  the  DWMS  panel  was  comprised  of  members  with  a 
wide  range  of  experience  in  the  ASD  POM  process,  an  interview  was 


designed  and  conducted  in  order  to  discuss  the  panel  members'  views 
about  the  1982-1986  POM  process.  The  questions  were  designed  in 
order  to  refresh  the  members'  memories  about  the  procedures  the 
DWMS  panel  used  in  evaluating  and  priority  ranking  decision  packages, 
to  discuss  the  pros  and  cons  of  the  procedures  used,  and  to  discuss 
recommended  improvements  in  future  POM  evaluation  processes .  It 
was  important  to  evaluate  whether  the  panel  members  understood  the 
philosophy  and  purpose  of  ZBB  and  the  POM  ranking  process  and  the 
importance  of  the  POM  to  not  only  ASD  but  also  the  whole  AF. 

The  interview  was,  also,  used  to  evaluate  the  degree  of  con¬ 
sensus  and  consistency  between  panel  members  with  respect  to  the 
purpose  of  the  panel,  the  evaluation  criteria,  and  the  procedures 
used  to  evaluate  and  rank  decision  packages.  Two  types  of  consis¬ 
tency  are  important:  internal  consistency  and  external  consistency. 
Internal  consistency  pertains  to  the  same  panel  member  evaluating 
all  decision  packages  in  the  identical  manner.  External  consistency 
addresses  the  question  of  whether  two  or  more  panel  members  evalu¬ 
ate  the  same  decision  package  in  the  identical  manner.  In  order  to 
obtain  a  precise  evaluation  of  all  decision  packages,  the  evaluation 
process  has  to  be  both  internally  and  externally  consistent. 

DWMS  Interview  Questions  and  Responses 


Nine  of  the  ten  DWMS  panel  members  were  interviewed  and 


their  answers  to  specific  questions  are  discussed  below.  The  first 
question  discusses  the  procedures  used  by  the  DWMS  panel  in  obtaining 
the  final  ranking  of  DWMS  decision  packages. 


| 


1.  What  is  your  recollection  of  the  steps  that  the  DWMS 
mission  area  panel  went  through  last  year  in  order  to  technically  rank 
decision  packages  for  ASD's  input  to  the  AF  POM? 

During  the  first  meeting  after  the  panel  members  were 
assigned  from  their  respective  organizations,  the  group  discussed 
the  ground  rules  for  the  evaluation  process,  the  proposed  evaluation 
criteria,  and  the  schedule  of  events.  This  procedural  review  was 
conducted  before  the  group  began  to  review  the  decision  packages  and 
was  helpful  in  increasing  group  internal  and  external  consistency. 

New  members  were  especially  appreciative  of  this  meeting  because 
they  had  received  no  formal  training  in  either  ZBB  or  the  ASD  POM 
process.  This  initial  meeting  was  the  first  contact  that  several  of  the 
new  members  had  had  with  ZBB  and  the  ASD  POM  process. 

The  evaluation  criteria  from  the  1981-1985  POM  cycle  were 
reviewed  and  improved.  Consensus  of  opinion  was  obtained  on  the 
definition  of  the  resulting  evaluation  criteria  and  how  each  decision 
package  would  be  evaluated.  All  members  of  the  group  were  satisfied 
with  the  final  evaluation  criteria  and  evaluation  process.  One  member 
stated  that  there  may  have  been  other  ways  to  evaluate  the  decision 
packages  but  this  method  will  suffice.  Morris  states  that  in  making 
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policy  decisions  managers  don't  seek  to  maximize  or  minimize  but 
seek  to  "satisfice"  (Morris,  1977:588).  The  final  evaluation  criteria 
are  shown  in  Figure  6. 

.  Importance 

.  Effectiveness 

.  Mission  Accomplishment 
.  Survivability 
.  System  Improvement 
.  Efficiency/Economy 

.  Life  Cycle  Costs 
.  Standardization 
.  Training 

.  Uniqueness 

.  Reality 

Figure  6:  DWMS  Program  Evaluation  Criteria 
(Weber,  1980a) 

The  scoring  model  methodology  that  the  panel  would  be  using 
was  reviewed.  Each  panel  member  would  subjectively  evaluate  each 
decision  package  as  to  its  contribution  to  AF  goals  using  the  evalu¬ 
ation  criteria  effectiveness ,  efficiency/ economy,  uniqueness,  and 
reality.  The  ordinally  scaled  benefits  for  each  criteria  ranged  from 
one  (high)  to  three  (low).  An  individual's  total  score  for  one  decision 
package  was  obtained  by  summing  the  individual's  scores  for  each 
criterion.  Then,  all  ten  member's  individual  total  scores  were 
summed  in  order  to  obtain  the  total  benefit  of  that  decision  package  to 
the  AF. 

An  implicit  assumption  made  by  the  DWMS  panel  was  that  in 


determining  the  total  benefits  of  each  decision  package,  all  decision 
packages  were  considered  independent  R&D  projects.  The  fault  with 
this  assumption  is  that  all  decision  packages  for  the  same  program 
element  are  interdependent.  ASD/ACB  letter,  16  Nov  1979,  "FY  82- 
86  POM  Submission,”  requires  all  decision  packages  to  be  written  bn 
an  incremental  basis  (ASD/ACB,  1979:49),  i.e.,  DP2  includes  DPI  and 
DP3  includes  DP2  and  DPI.  Therefore,  the  benefits  to  the  AF  from 
each  decision  package  should  be  evaluated  on  an  incremental  basis. 
What  additional  benefits  to  the  AF  are  incurred  by  selecting  DP2  over 
DPI  or  DP 3  over  DP2? 

Finally,  during  the  first  meeting,  the  panel  decided  that  each 
program  manager  should  be  allowed  to  present  a  short  briefing  in 
order  to  defend  his  program  and  to  answer  questions.  Several 
important  factors  entered  into  this  decision.  First,  the  time  in  which 
to  evaluate  approximately  seventy  decision  packages  was  extremely 
limited.  Second,  four  out  of  the  ten  members  were  new  to  the  POM 
ranking  process.  Finally,  each  panel  member  had  expertise  only  in 
the  projects  from  that  member's  organization.  By  having  the  pro¬ 
gram  managers  available  to  defend  their  program,  any  questions  that 
were  not  answered  by  the  decision  packages  could  be  answered  by  the 
program  manager. 

At  the  end  of  the  first  meeting,  the  panel  members  were 
given  a  copy  of  all  the  decision  packages  within  the  DWMS  mission 
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area.  During  the  next  two  days,  they  had  to  review  all  the  decision 
packages  and  prepare  questions  for  the  program  managers.  After 
the  program  managers  discussed  their  programs  and  answered  the 
panel's  questions,  the  panel  members  had  another  two  days  to  evalu¬ 
ate  all  decision  packages  and  determine  a  total  score  for  each  decision 
package  using  the  scoring  model  previously  discussed. 

When  the  complete  DWMS  panel  reconvened,  the  total  bene¬ 
fit  for  a  decision  package  was  calculated  by  summing  each  panel 
member's  individual  total  score  for  that  decision  package.  Any  large 
inconsistencies  as  shown  by  a  large  variance  in  individual  total 
scores  for  a  particular  decision  package  were  discussed  so  that  the 
members  could  be  in  complete  agreement  with  the  final  evaluation. 
Then,  the  initial  decis ion  package  ranking  was  obtaining  by  ordering 
all  the  packages  according  to  their  total  benefits.  At  this  time,  the 
interrelationship  between  decision  packages  for  the  same  program 
element  was  considered.  If  DP2  was  ranked  higher  than  DPI,  then 
DPI  was  dropped  off  the  list  since  DP2  included  DPI.  Similarly, 
if  DP3  appeared  above  DP2  and  DPI,  then  both  DP2  and  DPI  were 
dropped. 

Next,  using  the  intuition  of  the  panel  members,  the  initial 
ranking  was  reviewed  to  determine  if  it  seemed  to  be  "reasonable" 
to  all  the  panel  members.  ASD  overhead  DP3,  4950th  Test  and 
Evaluation  Squadron  overhead  DP 3,  and  ASD  telecommunications 
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DP3,  although  in  the  "definitely  funded"  category  of  projects  but  not 
the  top  three  decision  packages ,  were  moved  to  the  top  three  positions. 

Once  the  panel  members  were  in  complete  agreement  on  the 
priority  structure  of  the  decision  packages,  the  estimated  funding 
level  was  included  in  the  analysis.  This  figure  is  either  the  summa¬ 
tion  of  all  the  DP3's  for  the  approved  programs  in  1982  from  the  Five 
Year  Defense  Plan  or  is  provided  by  higher  headquarters  (ASD/ACB, 
1979:1).  Next,  the  cumulative  cost  total  for  the  decision  package 
priority  structure  was  calculated  starting  with  the  number  one  priority 
DP  and  continuing  through  the  last  DP.  All  decision  packages ,  whose 
cumulative  total  cost  was  below  the  budget  figure,  were  in  contention 
for  recommendation  to  the  Program  Review  Group  for  being  included 
in  ASD's  input  to  the  POM. 

Before  the  final  portfolio  was  recommended,  the  DWMS  panel 
reviewed  the  programs  which  were  just  above  and  just  below  the  bud¬ 
get  line.  The  purpose  of  this  review  was  to  determine,  based  upon  the 
panel's  intuition,  if  a  better  portfolio  could  be  obtained  by  moving  one 
project  below  the  funding  line  and  several  smaller  projects  above  the 
line.  After  the  DWMS  panel  arrived  at  a  final  consensus  on  the  prior¬ 
ity  structure  for  the  decision  packages,  this  ranking  was  recommended 
to  the  Program  Review  Group  for  their  review  and  evaluation. 

During  the  initial  interview  with  the  DWMS  panel  chairman, 
he  stated  that  the  POM  process  is  a  "technical  ranking"  of  decision 
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packages  (Weber,  1980a).  In  order  to  evaluate  the  degree  of  consensus 
within  the  DWMS  mission  area  panel  to  the  definition  of  "technical 


ranking,"  the  following  question  was  asked. 

2.  What  is  your  interpretation  of  the  words  "technical 
ranking"  as  used  to  describe  the  POM  ranking  process? 

All  of  the  panel  members  agreed  that  the  POM  ranking  pro¬ 
cess  was  not  a  pure  technical  ranking;  the  major  question  was  not 
whether  the  decision  package  could  be  technically  achieved  but  rather 
how  much  does  this  decision  package  contribute  to  satisfying  AF  needs 
and  deficiencies.  The  question  asked  by  DWMS  panel  members  was, 
how  would  I  rank  this  decision  package  against  all  others  in  this  mis¬ 
sion  area  based  upon  its  contribution  to  the  objectives  of  the  AF  both 
now  and  in  the  future? 

Agreement  was  not  obtained  on  whether  project  cost  should 
or  should  not  be  considered  in  defining  "technical  ranking.”  One 
member  stated  that  the  initial  ranking  should  be  conducted  without 
considering  the  cost  of  the  programs.  Another  member  stated  that 
the  cost  of  a  program  is  very  important  and  should  be  considered 
because  one  very  good  program  may  require  the  majority  of  the 
available  funds.  Twiss  comments  that  the  whole  R&D  portfolio  needs 
to  be  maximized  which  may  result  in  excluding  a  high  value  project 
because  of  its  high  consumption  of  the  available  resources  (Twiss, 
1974:37).  Another  member  mentioned  that  ail  projects  should  be 
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ranked  using  benefit-cost  analysis.  Finally,  one  member  stated  that 
return  on  investment  (ROI)  should  be  used  to  help  rank  programs. 

The  problem  with  the  use  of  ROI  is  that  some  of  the  "level  of  effort" 
programs  required  a  minimum  ROI  of  20  to  1  before  being  undertaken 
while  the  ROI  for  other  program  elements  would  be  impossible  to 
determine. 

The  following  six  questions  were  aimed  at  an  in  depth  analy¬ 
sis  of  each  member's  understanding  of  the  purpose  of  the  DWMS 
mission  area  and  the  objectives  of  the  DWMS  programs,  of  the  avail¬ 
able  information  about  each  decision  package,  and  of  the  evaluation 
criteria  and  other  influential  factors. 

3.  What  is  your  interpretation  of  the  purpose  and  objectives 
of  the  DWMS  mission  area? 

The  purpose  of  the  DWMS  mission  area,  according  to  the 
panel  members,  was  to  provide  a  mission  area  for  those  programs 
that  support  several  other  mission  areas  or  whose  descriptions  do  not 
conform  to  the  definitions  of  programs  contained  in  the  other  mission 
areas.  Some  very  small  projects,  which  were  needed  in  order  to 
support  the  objectives  of  the  combat  mission  areas  but  were  too  small 
to  compete  for  funds  alone,  were  combined  with  other  similar  pro¬ 
jects  into  one  program  element  which  was  competitive  within  the 
DWMS  mission  area.  Since  many  of  the  projects  support  many  or  all 
of  the  other  mission  areas,  it  would  not  be  practical  to  breakup  the 
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projects  so  that  a  portion  would  be  funded  by  the  mission  area  which 
it  supported.  One  panel  member  commented  that  some  of  the  DWMS 
programs  provided  needed  capabilities  that  would  not  be  funded  in 
another  mission  area.  Another  panel  member  commented  that 
several  programs  were  placed  in  the  DWMS  mission  area  because 
higher  management  felt  that  these  programs  could  compete  more 
easily  for  funds  in  the  DWMS  mission  area. 

4.  Was  it  clear  how  the  DWMS  decision  packages  contri¬ 
buted  directly  to  satisfying  Air  Force  needs  and  deficiencies,  or  if 
not  directly,  then  indirectly,  through  support  of  programs  in  other 
mission  areas? 

All  panel  members  stated  that  it  was  not  clear  hew  the  DWMS 
decision  packages  contributed  to  satisfying  AF  needs  and  objectives. 
The  members  had  to  rely  on  their  intuition,  experience,  and  sub¬ 
jective  judgements  in  order  to  relate  the  DWMS  programs  to  AF 
goals.  One  member  stated  that  the  problem  was  caused  by  the  DWMS 
programs  having  only  second  and  third  order  relationships  to  AF 
goals  by  aiding  programs  in  other  mission  areas  achieving  the  goals 
of  those  mission  areas.  Several  members  commented  that  a  short 
briefing  on  the  goals  and  objectives  of  ASD,  AFSC,  and  the  AF  over 
the  next  fifteen  years  as  well  as  the  SecDef's  direction  for  the  AF 
would  aid  in  the  decision  process .  McCandless  states  that  the  initial 
step  in  the  ZBB  process  is  for  top  management  to  provide  lower 


management  with  the  strategic  plan  of  the  organization  and  the 
assumptions  used  in  deriving  that  plan  (McCandless,  1978:46).  The 
problem  at  ASD  was  that  although  AF  and  DOD  do  provide  planning 
documentation,  the  documentation  is  either  not  readily  available;  to 
voluminous  to  be  analyzed  in  the  short  time  available  for  the  POM 
ranking;  or  as  in  the  DWMS  mission  area  case,  provides  only  scat¬ 
tered  and  partial  information  relating  projects  to  AF  needs. 

Several  members  commented  that  many  decision  packages 
were  written  so  that  the  objectives  of  those  decision  packages  were 
not  related  to  deficiencies  of  the  AF.  Another  comment  was  that  the 
objectives  of  a  decision  package  were  written  with  respect  to  the  AF 
needs  when  the  project  was  started,  not  the  current  needs  of  the  AF. 
The  decision  packages'  objectives  should  be  related  to  the  strategic 
goals  of  the  organization.  ASD/ACB  letter,  dated  16  November  1979, 
stated  that  the  long  range  capability  goals  of  each  decision  package 
should  be  directed  towards  the  general  need  of  the  AF  (ASD/ACB, 
1979:32).  Letzkus  states  that  the  objectives  of  a  program  must  be 
continually  evaluated  because  of  the  dynamic  nature  of  the  AF  environ¬ 
ment  causing  AF  needs,  requirements,  and  objectives  to  change  over 
time  (Letzkus,  1978:38).  All  of  the  panel  members  stated  that  the 
program  manager's  briefings  were  extremely  important  because 
questions  concerning  the  individual  programs  could  be  answered  at 
this  time.  Finally,  one  member  stated  that  if  the  decision  packages 
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for  a  program  were  clearly  related  to  AF  needs,  it  was  easy  to  eval¬ 
uate  the  program's  value.  It  appears  that  a  briefing  to  the  program 
managers  concerning  AF  goals  and  objectives  before  the  decision 
packages  were  prepared  would  help  to  standardize  the  information 
contained  in  the  decision  packages. 

5.  To  aid  in  evaluating  each  decision  package  with  respect 
to  AF  goals,  what  additional  type  of  input  information  would  you  have 
liked  to  have  had  available  and  would  like  to  see  in  the  future? 

All  members  stated  that  they  would  have  liked  to  have  avail¬ 
able  the  documents  which  authorized  the  programs.  The  Statement  of 
Operational  Need,  the  Mission  Element  Needs  Statement,  or  the  Pro¬ 
gram  Management  Directive  would  provide  additional  information 
justifying  each  program.  A  better  definition  of  the  requirements  that 
the  decision  package  were  suppose  to  satisfy  as  well  as  a  brief  sum¬ 
mary  of  how  this  decision  package  contributed  to  the  current  direction 
of  the  Air  Force  would  aid  the  panel  in  its  evaluation  process.  One 
panel  member  stated  that  several  decision  packages  seemed  more 
like  marketing  brochures  rather  than  a  concise,  technical  description 
of  where  the  program  is  and  where  it  is  going.  Although  Burroughs 
says  that  each  decision  package  should  include  performance  measures 
for  evaluating  the  effectiveness  and  efficiency  of  the  proposal  (Bur¬ 
roughs,  1977:8),  Suver  and  Brown  state  that  for  public  programs,  such 
as  national  defense,  output  measurements  in  terms  of  achievement 
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are  either  too  subjective,  lacking,  or  too  argumentative  (Suver  and 
Brown,  1977:81).  It  is  very  difficult  to  measure  the  output  in  terms 
of  increased  national  security  from  a  DWMS  program  and  more  pro¬ 
gram  documentation  is  not  the  solution.  Several  members  commented 
on  the  fact  that  they  could  not  read  and  absorb  all  the  information 
already  provided  in  the  decision  packages  and  relied  mainly  on  the 
Decision  Unit  Overview  and  Decision  Unit  Summary  forms  for  infor¬ 
mation.  Increasing  the  guidance  and  providing  training  to  the  program 
managers  on  decision  package  preparation  would  increase  the  quality 
and  consistency  of  the  decision  packages  which  would  provide  all  the 
required  information  for  effective  evaluation. 

6.  With  respect  to  the  input  information  that  was  available, 
what  factors  did  you  consider  the  most  important? 

Concerning  the  evaluation  criteria,  all  panel  members  felt 
that  effectiveness  was  the  most  important  followed  by  efficiency/ 
economy  although  in  the  scoring  model  all  evaluation  criteria  were 
weighted  equally.  Effectiveness  is  concerned  with  the  accomplishment 
of  stated  objectives  while  efficiency  assess  these  accomplishments  in 
terms  of  the  resources  used.  Effectiveness  is  especially  significant 
in  the  military  R&D  project  selection  decision  where  a  profit  motive 
is  lacking  because  the  ultimate  test  for  the  success  of  a  project  is  the 
ability  to  accomplish  the  required  mission. 

Several  members  mentioned  that  the  program  manager's 


briefings  were  invaluable  because  this  was  an  opportunity  to  ask 
questions  in  order  to  gain  information,  not  provided  in  the  decision 
packages,  about  the  programs.  Finally,  one  panel  member  mentioned 
the  political  realities  involved  with  a  particular  program.  It  did  not 
make  sense  to  recommend  a  program  to  be  funded  knowing  that  it 
would  not  be  supported  by  higher  management. 

7.  a.  What  is  your  interpretation  of  each  evaluation  criteria? 

b.  Do  you  feel  that  this  set  of  evaluation  criteria  was  suf¬ 
ficient  to  realistically  evaluate  each  decision  package  with  respect  to 
its  ability  to  meet  projected  Air  Force  needs?  If  not,  what  evaluation 
criteria  would  you  recommend1 

c.  What  type  of  questions  do  you  feel  effectively  evaluate 
each  decision  package  with  respect  to  the  listed  criteria? 

Importance  was  considered  to  be  comprised  of  two  subcri¬ 
teria,  effectiveness  and  efficiency/economy .  Effectiveness  pertains 
to  mission  accomplishment,  which  may  be  the  most  important  consid¬ 
eration  in  a  military  environment.  Also,  considered  under  effective¬ 
ness  were  the  survivability  of  the  system  in  a  combat  environment, 
and  the  question,  does  this  project  contribute  to  the  improvement  of 
a  combat  weapon  system?  Effectiveness  was  described  by  the 
question,  how  well  does  the  decision  package  fulfill  its  objectives, 
which  are  related  to  AF  needs? 

Efficiency/economy  evaluates  the  life  cycle  cost, 


standardization,  and  training  requirements  of  the  decision  package. 
The  questions  asked  with  respect  to  efficiency/economy  were,  given 
that  the  decision  package  is  effective,  is  the  job  being  accomplished 
for  the  least  cost,  or  does  the  package  contribute  to  the  standardiza¬ 
tion  of  other  programs?  With  Congressional  interest  on  military 
spending,  reducing  system  life  cycle  cost  and  increasing  standardi¬ 
zation  between  systems  is  receiving  greater  importance  in  project 
evaluations . 

The  following  is  a  list  of  questions  which  the  panel  members 
used  to  evaluate  the  uniqueness  of  a  program.  Given  that  the  program 
is  important,  is  the  DWMS  mission  area  the  only  funding  source  for 
the  program?  Is  this  project  the  only  effort  being  conducted  by  DOD 
to  satisfy  a  requirement  or  correct  a  deficiency?  Is  this  a  "one  of  a 
kind"  AF  project  which  is  needed,  but  if  it  is  not  funded  in  the  DWMS 
mission  area,  it  will  not  be  accomplished?  Is  any  other  DOD  agency 
performing  the  same  work  or  is  the  DWMS  mission  area  the  only 
mission  area  in  which  this  work  is  performed?  Finally,  is  this  a 
joint  agency  project  where  there  exists  a  "letter  of  agreement"  be¬ 
tween  the  AF  and  another  federal  agency  which  states  that  the  AF  will 
provide  a  certain  level  of  funds?  Rather  than  being  considered  as  an 
evaluation  criteria,  uniqueness  could  be  included  in  an  R&D  project 
selection  model  as  a  compulsory  project  with  a  lower  bounded 
decision  package. 
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The  last  evaluation  criteria,  reality,  evaluated  how  realistic 
the  project  is  in  the  current  political,  economic,  technical,  and  threat 
environments.  A  very  important  factor  in  evaluating  any  project  is 
the  political  support  for  the  project.  Who  is  the  champion  of  the  pro¬ 
ject?  Does  the  project  have  organizational  support?  Based  upon  each 
member's  intuition,  does  the  decision  package  documentation  clearly 
justify  the  project  so  that  it  has  a  high  probability  of  being  funded? 

If  a  project  is  dropped  at  higher  management  levels,  those  funds  may 
be  lost  because  another  project  may  not  be  substituted. 

Several  members  mentioned  the  importance  of  considering 
the  technical  feasibility  and  risk  of  the  project.  Risk  occurs  due  to 
the  uncertain  nature  of  the  outcome  of  an  R&D  project  (Seiler,  1965: 
129)  and  due  to  the  subjective  nature  of  the  input  parameters  used  to 
evaluate  an  R&D  project  (Blake,  1978:57).  What  is  the  probability  that 
the  project,  if  funded,  will  be  a  success?  Is  the  project  based  upon 
current  state  of  the  art,  or  does  it  depend  upon  the  technological 
efforts  in  a  laboratory?  Time  and  effort  are  wasted  in  evaluating 
projects  for  inclusion  in  the  budget  if  there  is  a  high  probability  or 
certainty  that  the  underlying  technology  will  not  be  available  from  the 
laboratory  for  several  years  or  if  the  laboratory  does  not  plan  to  fund 
the  required  technology.  The  program  managers  should  include  in 
their  decision  package  submissions  any  changes  that  have  occured  in 
the  initial  estimates  of  when  technology  will  be  available  from  the 
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laboratories.  The  dependency  of  a  DWMS  project  on  laboratory 
technology  can  be  included  in  an  R&D  project  selection  model  as  a 
mutually  dependent  project  constraint  with  the  laboratory  program 
element  considered  a  mandatory  proj  ect  if  selected  by  the  laboratory's 
ranking  process. 

Several  members  stated  that  cost  versus  time  and  cost 
versus  payoff  were  important  considerations  when  evaluating  the 
reality  of  a  project.  Is  the  project  schedule  realistic?  Can  the  pro¬ 
ject  be  completed  in  the  stated  time  for  the  stated  cost,  or  is  there 
a  high  probability  that  the  program  will  slip  resulting  in  cost  over¬ 
runs?  Are  the  future  costs,  as  presented  in  the  decision  packages, 
realistic?  Will  funding  a  project  now  result  in  a  large  increase  in 
funds  required  in  the  future?  Do  the  benefits  received  from  the  pro¬ 
ject  justify  the  cost,  or  should  the  funds  be  used  to  support  another 
project  which  has  a  higher  contribution  to  the  goals  of  the  AF? 
Harrington  discusses  a  decision  tree  format  which  includes  the  multi¬ 
period  aspect  of  R81D  projects  in  a  mathematical  programming  model 
(Harrington,  1979:25).  The  five  year  funding  requirements  for  a 
project  as  expressed  in  the  FYDP  can  be  formulated  as  constraints 
in  a  selection  model. 

Lastly,  several  members  mentioned  the  possibility  that  a 
project  could  be  tied  to  a  multiyear  contract.  If  the  project  is 
required  to  be  funded  at  a  stated  level,  a  compulsory  project  with  a 
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lower  bound  decision  package  constraint  can  be  included  as  a  part  of 
a  selection  model. 

8.  Concerning  the  overall  POM  ranking  process,  what  type 
of  external  pressures  and  factors  were  present  and  should  be  included 
in  the  evaluation  process? 

Most  of  the  panel  members  remarked  that  external  influences 
were  minimal  to  non-existent  throughout  the  evaluation  process.  The 
panel  did  not  try  to  second-guess  higher  management  and  tried  to 
provide  their  best  judgement  on  the  programs  with  full  justification 
for  their  approval  or  non-approval  of  any  program.  One  member  did 
state  that  external  influences  were  included  in  the  evaluation  process 
under  the  evaluation  criteria,  reality.  Another  member  said  that  it 
was  impossible  to  operate  in  a  vacuum  and  not  be  influenced  by 
external  factors  especially  when  it  was  general  knowledge  how  top 
management  feels  about  certain  DWMS  projects.  According  to  the 
philosophy  of  ZBB,  each  level  of  management  is  supposed  to  evaluate 
each  project  based  upon  its  contribution  to  organizational  goals  and 
not  upon  lower  management's  perceived  desires  of  top  management. 
Top  management  provides  the  long  range  goals  of  the  organization, 
and  lower  management  provides  a  project  priority  ranking  based 
upon  their  evaluation  of  each  project's  contribution  to  those  organi¬ 
zational  goals. 

The  next  three  questions  discuss  the  DWMS  panel  members' 
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views  on  the  major  problem  areas  encountered,  strengths  and  weak¬ 
nesses  of  the  ranking  process  that  the  DWMS  panel  used,  and  recom¬ 
mendations  for  the  future. 

9.  With  respect  to  the  overall  objectives  of  the  DWMS  panel, 
what  were  the  major  problems  encountered  in  the  ranking  process? 

The  major  complaint  from  the  panel  members  was  that  there 
were  too  many  decision  packages  to  properly  review  and  evaluate  in 
the  amount  of  time  available.  Only  about  a  week  and  one  half  was 
available  to  review,  evaluate,  rank,  and  justify  about  seventy  deci¬ 
sion  packages.  The  panel  members  that  had  never  been  involved  with 
the  POM  ranking  process  said  that  the  process  was  extremely  difficult 
for  them  because  they  not  only  had  to  review  all  the  decision  packages 
but  also  had  to  learn  the  POM  process  philosophy  before  an  adequate 
job  in  ranking  packages  could  be  performed.  All  members  stated 
that  more  time  was  needed  in  order  to  fully  evaluate  all  the  decision 
packages  and  to  make  accurate  tradeoffs  between  them.  Anthony 
(Hyde,  1978: 322)  and  McLaughlin  et  al.  (McLaughlin  et  al.  ,  1980:2) 
suggest  that  the  program  '-valuation  process  could  take  place  more 
effectively  outside  the  budget  cycle  when  there  does  not  exist  a  heavy 
workload  and  deadlines. 

10.  What  do  you  feel  were  the  strengths/weaknesses  of  the 
approach  that  the  DWMS  panel  used  to  technically  rank  decision 
packages? 
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The  major  strength  mentioned  by  the  panel  members  was 
the  efficiency  with  which  the  DWMS  panel  was  run.  The  administra¬ 
tive  procedures  were  very  well  preplanned  and  little  time  was  wasted 
before  the  actual  evaluation  process  began.  One  panel  member  did 
comment  that  a  better  definition  of  the  evaluation  criteria  would  have 
been  helpful  because  valuable  time  was  used  to  discuss  and  define  the 
criteria. 

Three  major  weaknesses  were  noted  by  the  members,  all  of 
which  dealt  with  the  external  environment  imposed  on  the  panel  by 
ASD  rather  than  the  procedures  used  by  the  DWMS  panel.  First,  as 
previously  discussed,  there  was  a  very  limited  amount  of  time  avail¬ 
able  in  which  to  evaluate  the  decision  packages.  Many  members  felt 
that  the  review  process  was  not  as  thorough  as  it  should  have  been 
because  of  the  time  constraint.  Second,  the  decision  packages  did  not 
contain  enough  detailed  information  by  which  adequate  tradeoffs  could 
be  determined.  Finally,  there  was  a  high  variance  in  the  quality  of 
information  contained  in  the  decision  packages  which  made  it 
extremely  difficult  to  evaluate  entirely  different  types  of  projects. 

12.  What  improvements  would  you  like  to  see  incorporated 
into  the  ASD  POM  process  in  order  to  improve  the  quality  of  the 
output? 

The  first  recommended  improvement  stated  by  ail  of  the 
members  was  to  increase  the  amount  of  time  available  to  review  the 
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decision  packages.  Since  the  people  at  ASD  know  that  the  POM 
ranking  process  occurs  every  year  and  during  the  same  time  period, 
the  program  managers  could  complete  their  decision  packages  earlier 
which  would  allow  the  Mission  Area  Review  panels  to  convene  sooner 
and  have  more  time  to  thoroughly  review  the  packages.  Although 
final  guidance  on  the  preparation  of  the  decision  packages  will  not 
have  been  received  by  ASD,  the  Mission  Area  Review  panels  would  at 
least  have  enough  information  to  begin  the  review  process. 

Another  member  stated  that  a  better  understanding  of  how 
each  decision  package  contributes  to  the  accomplishment  of  AF  goals 
and  objectives  would  aid  in  the  ranking  process.  The  goals  and 
objectives  for  many  decision  packages  were  stated  in  terms  of  what 
that  package  was  suppose  to  achieve,  not  how  that  package  aids  in 
eliminating  the  deficiencies  in  the  combat  mission  areas.  The  deci¬ 
sion  packages  need  to  be  related  directly  to  the  goals  of  the  AF. 

Many  members  stated  that  the  tradeoff  decisions  among  the 
completely  different  types  of  programs  included  in  the  DWMS  mission 
area  could  have  been  improved  if  the  decision  packages  were  written 
more  consistently.  The  quality  and  type  of  information  contained  in 
the  decision  packages  varied  widely  and  caused  considerable  difficulty 
in  the  evaluation  and  tradeoff  decision  process.  Several  members 
recommended  that  a  good  quality  control  system  be  installed  to 
review  the  decision  packages  before  the  DWMS  panel  receives  them 
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for  evaluation.  Suver  and  Brown  discuss  the  problems  that  can  occur 
when  management  tries  to  compare  the  outputs  from  non-related 
decision  packages  (Suver  and  Brown,  1977:81).  By  definition,  the 
DWMS  mission  area  contains  many  non-related  programs. 

All  of  the  members  stated  that  they  would  like  to  have  had 
some  type  of  feedback  on  how  their  ranking  compared  to  the  rankings 
of  higher  management.  Without  any  information  as  to  how  the  rankings 
changed  at  higher  management  levels,  many  members  questioned  the 
value  of  the  time  spent  reviewing  and  ranking  the  decision  packages. 
McCandless  states  that  as  the  last  step  in  the  ZBB  process  top 
management  should  provide  all  lower  levels  of  management  the  final 
ranking  and  any  reasons  for  changes  to  the  initial  rankings.  In  this 
way,  lower  management  will  build  up  trust  in  top  management  and 
provide  complete  and  unbias  information  in  the  future  (McCandless, 
1978:46-47). 

Finally,  one  member  said  that  there  was  no  reward  or 
recognition  for  working  on  the  DWMS  panel.  The  POM  ranking  pro¬ 
cess  was  a  short  two  week  project  that  detracted  from  the  panel 
members'  primary  jobs  with  their  organizations.  They  were  valuable 
people  from  the  organization  which  they  represented  and  their  immed¬ 
iate  superiors  would  rather  have  had  them  working  on  their  projects 
than  on  the  POM.  The  ZBB  process,  when  initiated  within  the  DOD, 
represented  major  change  to  the  organization.  According  to 
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McCandless,  major  change  in  an  organization  involves  three  steps. 
First,  the  old  ideas  must  be  unfrozen.  Next,  the  old  ideas  must  be 


changed  to  the  new  ideas.  Finally,  the  new  ideas  must  be  refrozen 
and  supported  by  the  organization's  reward  system  or  else  the  organ¬ 
ization's  people  will  have  a  tendency  to  revert  to  the  old  ideas 
(McCandless,  1978:47).  The  ASD  reward  system  as  perceived  by 
several  panel  members  did  not  support  the  panel  members'  work. 


This  chapter  provided  a  brief  description  of  the  DWMS  mis¬ 
sion  area  and  its  programs  and  summaried  ideas  concerning  the  ASD 
POM  process  obtained  by  interviewing  the  DWMS  panel  members. 
Based  upon  these  ideas  and  concepts  discussed  in  the  literature 
review,  a  methodology  will  be  developed  in  the  following  chapter, 
which  the  DWMS  panel  could  use  to  aid  in  future  evaluation  and  selec¬ 
tion  of  DWMS  programs  for  the  POM. 
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Chapter  5 


DWMS  PROJECT  EVALUATION/ 

PROJECT  SELECTION  MODEL 

Once  the  DWMS  Mission  Area  Review  panel  has  received  all 
the  decision  package  documentation  from  the  program  managers, 
they  have  to  review,  evaluate,  and  rank  all  those  decision  packages 
according  to  their  contribution  to  AF  needs.  Since  the  determination 
of  the  contribution  to  AF  needs  of  any  military  R&D  program  requires 
a  very  subjective  judgement  by  the  evaluator,  a  process  that  reduces 
evaluation  inconsistencies  and  biases  would  be  beneficial  to  this  R&D 
project  selection  process .  This  chapter  discusses  a  scoring  model 
that  the  DWMS  panel  can  use  to  evaluate  programs  and  to  determine 
their  benefits  to  the  AF.  Once  the  benefits  of  each  decision  package 
and  incremental  benefit  adjustments  due  to  decision  package  inter¬ 
relationships  have  been  determined,  a  0-1  integer  programming 
model  is  used  to  select  that  project  portfolio  with  maximum  benefits 
to  the  AF.  This  chapter  includes  a  demonstration  of  the  0-1  IP  model 
applied  to  a  hypothetical  ZBB  problem. 

DWMS  Benefit  Measurement  Model 

Most  of  the  DWMS  panel  members  stated  that  a  scoring 
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model  similar  to  the  one  used  to  evaluate  decision  packages  for  the 
1982-1986  POM  would  be  satisfactory  in  evaluating  the  contributions 
of  each  decision  package  to  the  goals  of  the  AF.  They  felt  that  a  more 
complicated  evaluation  model  would  not  be  appropriate  because  of  the 
highly  subjective  nature  of  the  input  data.  The  members  did  question 
whether  the  criteria  were  appropriate,  but  given  the  time  constraint, 
they  all  felt  that  the  model  they  used  was  adequate.  One  area  of 
improvement  recommended  by  the  panel  was  that  an  exact  definition 
of  each  evaluation  criteria  is  required.  Several  members  felt  that 
too  much  valuable  time  had  been  wasted  discussing  the  particulars  of 
each  criteria.  Even  after  the  discussions,  several  members  were 
still  uncertain  as  to  the  exact  definition  of  the  criteria  and  as  a  result, 
were  not  consistent  in  evaluating  all  seventy  decision  packages.  The 
members  were  not  certain  whether  the  last  package  was  evaluated  in 
the  same  manner  as  the  first  package.  Also,  if  all  members  had  a 
slightly  different  definition  for  each  criteria,  then  they  were  in  reality 
using  different  criteria  to  judge  the  decision  packages.  A  more  explic¬ 
it  definition  for  each  evaluation  criteria  would  improve  both  the 
internal  and  external  consistency  of  the  evaluation. 

As  an  additional  aid,  besides  a  more  exact  definition,  several 
members  mentioned  that  a  list  of  typical  questions  that  should  be 
asked  for  each  criteria  would  assist  in  improving  the  consistency  of 
the  evaluation  process .  The  purpose  of  the  questions  would  be  to 
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stimulate  thought  rather  than  being  answered  specifically.  If  the 
questions  are  thorough,  then  the  panel  members  can  assume  that 
each  decision  package  is  evaluated  consistently  between  members  and 
that  each  member's  evaluation  of  the  first  and  the  last  decision  pack¬ 
age  is  equivalent. 

Based  on  the  interviews  with  the  DWMS  panel  members  and 
the  literature  review  into  R&tD  project  evaluation  techniques,  a 
scoring  model  was  selected  as  the  appropriate  model  for  the  DWMS 
panel  to  use  to  evaluate  decision  package  benefits.  This  type  of 
model  was  chosen  because  of  the  large  ratio  of  qualitative  to  quanti¬ 
tative  data  available,  the  high  uncertainty  in  the  R&D  environment, 
and  the  heavy  reliance  on  the  subjective  opinions  of  the  panel  mem¬ 
bers.  For  many  of  the  programs,  it  would  be  either  very  difficult, 
not  cost  effective,  or  too  time  consuming  to  collect  the  data  required 
to  quantify  the  benefits  for  either  a  net  present  value  or  internal  rate 
of  return  calculation.  Much  of  the  data,  if  it  could  be  collected, 
would  be  of  questionable  value  because  of  the  degree  of  uncertainty 
in  that  data.  The  panel  members  agreed  that  a  scoring  model  was 
the  most  appropriate  method  for  measuring  the  benefits  to  the  AF  of 
each  decision  package. 

Bemelmans  recommends  that  any  scoring  model  contain  no 
more  than  five  to  seven  main  criteria  because  it  becomes  very  diffi¬ 
cult  for  management  to  handle  a  large  number  (Bemelmans,  1979:39). 
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In  order  to  determine  the  appropriate  number  and  type  of  evaluation 
criteria  to  recommend  to  the  DWMS  panel,  three  sources  of  infor¬ 


mation  were  used.  First,  a  literature  review  was  conducted  into 
R&D  project  evaluation  techniques.  Although  most  of  the  information 
pertained  to  commercial  organizations,  many  of  the  evaluation  cri¬ 
teria  and  checklist  questions  could  be  adapted  to  military  R&D  pro¬ 
ject  selection.  Cetron  specifically  discusses  military  R&D  project 
selection  and  mentions  the  following  criteria: 

1)  military  utility, 

a)  value  to  warfare 

b)  task  responsiveness 

c)  timeliness 

2)  technical  feasibility,  and 

3)  financial  acceptability  (Cetron,  1969:57-60). 

Cooper,  in  discussing  R&D  project  evaluation  in  the  federal  govern¬ 
ment,  mentions  that  the  feasibility  that  a  project  will  be  successful 
depends  upon  the  technical  risk,  the  availability  of  appropriate  tech¬ 
nical  skills  and  facilities,  and  the  availability  of  management  talents 
(Cooper,  1978:31). 

The  second  source  of  information  was  from  interviews  con¬ 
ducted  with  the  Mission  Area  Review  panel  chairmen.  The  purpose 
of  the  interviews  was  to  discuss  the  methodology  that  their  panels 
used  to  evaluate  and  rank  decision  packages  within  their  mission 
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area.  One  evaluation  criteria  that  was  consistently  mentioned  by  the 
chairmen  was  that  of  program  timing.  Will  the  end  item  be  available 
at  the  appropriate  time  to  counter  the  threat?  A  typical  evaluation 
model  is  that  used  by  the  Air-to-Surface  Attack  Mission  Area  Review 
panel  which  consisted  of  the  following  criteria: 

1)  timeliness 

2)  risk 

a)  high  risk 

b)  medium  risk 

c)  low  risk 

3)  contribution 

a)  primary--contributes  directly  to  satisfying  the  need 

b)  secondary--contTibutes  directly  to  satisfying  a 
secondary  need 

c)  indirect--contributes  toward  allowing  another  system 
to  satisfy  a  need 

d)  no  contribution  (Matzko,  1980) . 

The  last  source  of  information  is  from  the  interviews  with 
the  DWMS  panel  members  themselves.  This  is  probably  the  most 
important  source  of  information  because  it  allows  the  information 
obtained  from  the  literature  and  the  other  mission  area  panel  chair¬ 
men  to  be  tailored  to  the  specific  requirements  of  the  DWMS  mission 
area.  After  reviewing  the  results  from  the  interviews,  especially 
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each  individual's  definition  of  the  evaluation  criteria,  it  became  ap¬ 
parent  that  five  factors  were  important  when  evaluating  DWMS  pro¬ 
grams.  These  factors  were  as  follows:  program  effectiveness  and 
mission  accomplishment,  program  feasibility,  resource  acceptability, 
program  timing,  and  program  importance. 

The  DWMS  panel  members  mentioned  that  program  effective¬ 
ness  was  the  most  important  evaluation  criterion  when  judging  a  pro¬ 
gram.  It  is  felt  that  effectiveness  and  mission  accomplishment  are 
part  of  a  broader  criterion,  military  value  or  relevance,  which 
attempts  to  measure  the  value  or  usefulness  of  the  program's 
expected  output  to  the  AF.  It  measures  whether  the  end  item  can 
satisfy  a  projected  military  need  with  respect  to  a  future  threat. 

Given  that  a  need  exists,  what  is  the  contribution  of  this  program  in 
the  military  environment  to  the  satisfaction  of  that  need?  Time¬ 
liness  is  an  integral  part  of  military  value  because  in  order  for  a 
project  to  be  useful,  it  must  provide  a  new  or  improved  capability 
in  the  shortest  possible  time  after  its  need  is  recognized.  In  order 
for  the  end  item  to  be  effective,  it  must  be  available  when  it  is  needed 
to  counter  the  threat.  When  evaluating  military  value,  the  decision 
maker  assumes  that  the  system  is  feasible  and  that  the  resources  are 
available  when  required.  Military  value  is  the  potential  value  from 
the  project,  and  in  order  to  determine  the  actual  value,  project 
feasibility  and  resource  acceptability  must  be  considered. 
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Program  feasibility  measures  the  risks  associated  with  the 
program  and  attempts  to  measure  how  realistic  the  decision  package 
is  in  the  current  political,  economic,  and  technical  environments.  If 
this  decision  package  is  approved,  what  is  the  probability  that  the 
stated  goals  will  be  achieved  with  the  resources  now  planned?  A  pro¬ 
gram  must  be  evaluated  with  respect  to  its  technical,  political,  and 
organizational  feasibility.  Technical  feasibility  measures  the  techni¬ 
cal  risks  associated  with  the  program.  It  is  a  measure  of  the  proba¬ 
bility  that  the  project  will  not  achieve  its  goals  due  to  technological 
bottlenecks.  Political  feasibility  measures  the  political  support  for 
the  program.  Who  is  the  program's  champion?  How  hard  is  the 
champion  pushing  the  program?  How  high  up  in  the  AF  did  the  pro¬ 
gram  originate?  The  higher  up  in  the  AF  that  the  project  can  gain 
attention  and  the  more  widespread  its  potential  impact,  the  greater 
the  incentive  for  its  resolution.  Organizational  feasibility  questions 
the  capability  of  the  organization  to  perform  the  work.  Is  the  organi¬ 
zation  technically  competent  to  perform  the  work?  Does  the  organiza¬ 
tion  have  the  required  engineering,  scientific,  and  managerial  talents 
to  successfully  complete  the  project?  Finally,  implicit  in  the  deter¬ 
mination  of  feasibility  is  program  timing.  As  an  example,  if  the  pro¬ 
gram  is  urgently  needed  and  as  a  result  the  program’s  schedule  is 
compressed,  then  the  technical  risks  associated  with  the  program 
could  be  increased  especially  if  advanced  technology  is  required. 
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Resource  acceptability  attempts  to  measure  the  reasonable¬ 
ness  of  the  resource  assumptions.  When  evaluating  project  resource 
requirements,  not  only  must  the  availability  of  the  required  resources 
be  determined  but  also  that  level  must  be  acceptable  to  the  organiza¬ 
tion.  For  example,  the  high  percentage  of  available  resources  con¬ 
sumed  by  one  decision  package  may  not  be  acceptable  to  the  DWMS 
mission  area  panel  because  it  would  starve  too  many  of  the  other 
programs.  If  this  decision  package  is  approved,  what  is  the  pro¬ 
bability  that  the  resources  planned  will  be  provided  considering  the 
competition  for  resources'5  Resource  timing  is  very  important  when 
determining  the  acceptability  of  the  resource  requirements  to  the 
DWMS  panel.  Will  the  resources  be  available  when  required  to 
successfully  complete  this  program?  Are  the  future  funding  require¬ 
ments  realistic,  and  if  questionable,  what  is  the  probability  of  a  cost 
overrun?  Resource  acceptability  can  only  be  accurately  determined 
by  evaluating  the  resource  requirements  over  the  complete  time 
period  in  the  FYDP. 

Just  determining  that  a  project  can  fulfill  a  need  is  not 
justification  for  allocating  scarce  AF  resources  to  the  fulfillment  of 
that  need.  The  additional  information  required  is  how  important 
satisfying  that  need  is  to  the  AF.  What  is  the  importance  of  the  AF 
need  being  addressed  with  respect  to  the  AF's  strategic  plan15  The 
AF  cannot  afford  to  satisfy  all  of  its  needs  and  therefore,  has 
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developed  a  priority  list  of  needs  which  provides  an  indication  of  the 
importance  that  the  AF  places  on  different  programs.  A  program 
may  satisfy  a  large  deficiency,  but  if  the  AF  does  not  place  much 
importance  on  the  elimination  of  that  deficiency,  then  the  program 
should  not  be  funded. 

Discussions  about  a  proposed  model  with  the  DWMS  panel 
chairman  in  order  to  help  verify  that  model  suggested  that  the  deter¬ 
mination  of  a  program's  importance  was  too  subjective  and  should  be 
handled  by  a  group  effort  (Weber,  1980b).  He  proposed  that  each  panel 
member  evaluate  a  decision  package  using  the  following  model: 

[" Individual"!  _  ^Military  Value  +  Feasibility  +  Resource  Accept-  • 

L  Score  J  ”  L  Score  Score  abilit  -  Score 

Once  the  panel  reconvenes,  a  group  importance  score  for  each  deci¬ 
sion  package  will  be  determined.  The  total  benefit  score,  which  will 
be  used  as  the  objective  function  coefficient  in  a  0- 1  IP  selection 
model,  will  be  the  sum  of  all  the  individual  scores  for  that  decision 
package  times  the  group  importance  score  for  that  decision  package . 
The  justification  for  multiplying  the  importance  score  is  that  if  a 
decision  package  is  rated  low  in  importance,  then  the  total  score  for 
that  decision  package  should  also  be  low.  The  scoring  model  is  shown 
in  Appendix  A. 

After  each  decision  package  has  been  individually  reviewed 
and  evaluated,  two  more  evaluations  have  to  be  performed.  First, 
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the  decision  packages  are  reviewed  to  determine  if  there  are  any 
compulsory  or  alternate  compulsory  packages.  If  there  are  any, 
those  packages  have  to  be  flagged  so  that  the  appropriate  constraints 
can  be  formulated.  The  second  review  is  to  determine  if  there  are 
any  interdependencies  between  programs .  Program  interdependencies 
will  be  formulated  in  the  form  of  a  self-interaction  matrix  which  will 
be  discussed  in  the  next  section  and  will  be  explicitly  included  in  the 
selection  model. 


Figure  7:  Binary  Interaction  Matrix 

Self-Interaction  Matrix 

To  assist  the  DWMS  panel  in  formulating  the  interrelation¬ 
ships  between  programs  but  not  within  programs,  an  adaptation  of 
the  self-interaction  matrix,  as  discussed  by  Sage,  will  be  used.  The 


self- interaction  matrix  is  a  convenient  format  for  presenting  the 


interrelationships  between  any  two  programs.  In  a  binary  interaction 
matrix.  Figure  7,  a  one  indicates  that  a  direct  interaction  between 
programs  occurs  while  a  zero  indicates  no  direct  interaction.  If 
there  are  n  elements,  there  are  entries  to  be  made  in  the 

self- interaction  matrix  (Sage,  1977:13-24). 

In  order  to  adapt  the  self- interaction  concept  for  use  by  the 
DWMS  panel,  several  changes  have  to  be  made.  First,  if  two  pro¬ 
grams  interact,  either  positively  or  negatively,  the  magnitude  of  that 
interaction,  expressed  as  an  adjustment  to  the  organization's  bene¬ 
fits,  is  entered  in  the  appropriate  matrix  pos ition.  Secondly,  the 
self- interaction  matrix  represents  interrelationships  between  pro¬ 
grams  but  not  within  programs .  Therefore,  the  matrix  elements 
which  correspond  to  an  interaction  between  decision  packages  for  the 
same  program  represent  non-feasibie  interactions.  Because  of  this 
last  requirement,  the  total  number  of  possible  interactions  is  given 
by  the  following  formula: 


r Total  Possible 
Interactions 


n  r-  n 

E  riv  '  E  m;  - 1  I 
i=l  iLi=i  j 


r-  n  HH-  1 

.r.  .e.  j  ! 

L  v=  1  j  =  1  1 


The  first  term  of  the  formula  is  an  adaptation  of  the  formula  given  by 
Sage  (Sage,  1977: 13)  to  the  case  of  ZBB  where  there  are  decision 
packages  in  each  of  i  programs.  The  second  term  is  an  adjustment 
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Figure  8;  Self-Interaction  Matrix  in  ZBB  Format 


to  disregard  those  elements  corresponding  to  decision  packages  for 
the  same  program  interacting  with  each  other.  Figure  8  illustrates 
the  self-interaction  matrix  adapted  to  ZBB  form. 

0-1  Integer  Programming  Model  Development 

Once  the  PWMS  panel  has  evaluated  each  decision  package 
and  determined  its  benefit,  that  benefit  can  be  used  as  the  coefficient 
of  the  objective  function  in  a  0-1  integer  programming  model.  A  0-1 
approach  has  been  selected  because  this  approach  allows  for  the  "go- 
no  go"  type  of  decision  required  in  ZBB.  Either  all  or  none  of  a 
particular  decis  ion  package  has  to  be  selected.  The  manner  in  which 
decision  packages  are  defined  and  constructed  does  not  allow  for  a 
fractional  part  of  a  decision  package  to  be  selected.  A  mathematical 
programming  approach  to  the  R&D  project  selection  decision  is  pre¬ 
ferred  to  either  scoring  models  or  economic  models  because: 

1)  it  can  be  formulated  to  include  multiple  resource  con¬ 
straints  over  several  time  periods; 

2)  it  can  be  formulated  to  include  compulsory  projects  and 
alternate  project  versions;  ;.nd 

3)  it  selects  a  portfolio  which  maximizes  total  benefits  to 
the  organization. 

Bell's  formulation  (Gear  et  al.  ,  1971:66-68)  of  the  R&D  pro¬ 
ject  selection  problem  has  been  adapted  to  the  ZBB  decision  package 
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selection  problem.  Improvements  to  Bell's  model,  which  allow  for 
project  interdependencies,  have  been  formulated.  Resource  con¬ 
straints  which  may  apply  to  the  DWMS  panel  are  multi-year  budget 
limits,  manpower  limitations,  and  facilities  contraints.  Additional 
possible  constraints  are  the  following:  a  compulsory  proj ect  with  a 
lower  bounded  decision  package,  alternate  compulsory  projects  with 
lower  bounded  decision  packages,  mutually  dependent  projects,  and 
mutually  exclusive  project  decision  packages. 

When  the  DWMS  panel  members  were  evaluating  each  deci¬ 
sion  package,  they  tried  to  consider  the  multi-year  funding  require- 
n.  nts  of  that  decision  package  and  the  impact  that  those  requirements 
would  have  on  future  project  portfolios.  Looking  only  at  the  first 
year  of  the  FYDP,  a  particular  program  could  have  a  low  funding 
requirement  and  seem  attractive  to  the  AF  but  the  funding  levels  in 
the  following  four  years  could  require  a  large  portion  of  the  esti¬ 
mated  budget.  Looking  at  the  complete  FYDP,  the  project  would  be 
very  unattractive  and  should  not  be  included  in  the  first  year's 
recommended  portfolio. 

Two  DWMS  projects  may  have  either  a  synergistic  or 
cannibalistic  effect  on  each  other,  and  their  summed  benefits  would 
not  reflect  their  true  contributions  to  the  DWMS  mission  area.  In 


order  to  include  project  interactions  in  the  0-1  IP  model,  the  objec- 


T 


tive  function  is  expanded  to  include  nonlinear  terms.  These  non¬ 
linear  terms  are  of  the  form,  xjjX2j,  and  signify  that  the  benefit 
contribution  to  the  organization  for  particular  combinations  of  any  two 
programs  is  not  additive.  The  coefficient  of  the  nonlinear  term  repre¬ 
sents  the  adjustment  to  the  objective  function  required  by  selecting 
a  particular  combination  of  projects  for  the  final  portfolio  and  is 
obtained  from  the  self-interaction  matrix. 

The  solution  method  depends  upon  the  approach  taken  to 
formulate  the  interaction  problem.  If  the  nonlinear  terms'  coeffi¬ 
cients  represent  a  change  in  benefits,  then  an  approach  to  solving 
nonlinear  0-1  programming  problems  as  suggested  by  McMillan  is 
adapted  to  the  ZBB  problem  (McMillan,  1975:499-501).  If  the  coeffi¬ 
cients  represent  the  total  benefits  to  the  organization,  then  the  prob¬ 
lem  can  be  formulated  and  solved  by  creating  a  new  project  which 
represents  the  combination  of  the  two  interacting  projects.  Both 
approaches  will  be  demonstrated  by  simple  examples. 

Zero-Base  Budgeting  0-1  Integer 
Programming  Model 

The  final  R&D  project  selection  model  suggested  for  use  by 
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the  DWMS  Mission  Area  Review  panel  is  of  the  form: 


n  m^  u 

Maximize:  z  =  £  £  b.-:  x;;  +  £  b_  1_ 

i=l  j  =  l  1]  s  =  l  s  s 


Subject  to: 

1.  Resource  Constraints 


n  mi 

E  E  a...  x..  $  Ai,_ 

i=l  j  =  l  ijfcp  ^3  kP 


k  a  1,2,  .  .  .  ,  N 
P  =  1.2,  ...,P 


2.  Mutually  Exclusive  Decision  Packages  for  the  Same 
Program 


mi 


E  x--  s  1 

jJl  lJ  ‘ 


i  ™  1,2,  f  •  * ,  n 


3.  Mutually  Dependent  Decision  Packages  for  Separate 
Programs 

a.  One-Way  Dependency:  If  at  least  decision  package 

x^r  is  selected,  then  at  least  decision  package 

x-  has  to  be  selected. 

4r4 

m  i  iru 

£  x,.  -  T?  x4i  3  0 

J*VJ  3=*4  ’ 

b.  Two-Way  Dependency:  If  either  at  least  decision 

package  x,  or  at  least  decision  package  x.  is 
ir 1  *r4 

and  at  least  x, 

1  4r4 


selected,  then  at  least  x,  and  at  least  x,  have 

lri  4r, 


to  be  selected. 
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m  |  m4 

E  x,,  -  E  x  .  =  0 

J  =  rl  1  i=r4  4j 


4.  Compulsory  Project  with  Lower  Bounded  Decision 
Package 


E  xy  =  1 


5.  Alternate  Compulsory  Projects  with  Lower  Bounded 
Decision  Packages 


m  i  m4 

t  *  ♦  r  *4j  ®  i 

J=rl  1  J=r4 


6.  Project  Interaction  Constraints 


x  ,  +  x  ,  -  I  SI 
ab  cd  s 

-x  -  x  +  21  SO 
ab  cd  s 


where  x  ,  and  x  ,  are 
ab  cd 

two  projects,  a  and  c, 
whose  decision  pack¬ 
ages,  b  and  d,  interact 
and  a  c . 


=  1  if  decision  package  j  of  project  i  is  selected 

=  0  if  decision  package  j  of  project  i  is  not 


selected 


l—  1,2,  •••,  n 


5  =  1»2,  '•«,  m. 


1 


8. 


Where: 


I  =  x  .  x  , 
s  ab  cd 


=  1  if  x  ,  and  x  ,  are  selected 
ab  cd 


=  0 

s  —  1 }  2 1  •  *  ■  f  u 


if  x  ,  and  x  ,  are  not  selected 
ab  cd 


b-. 
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is  the  benefit  of  decision  package  j  of  project  i 
is  the  incremental  benefit  change  caused  by 
selecting  both  xa^  and  xccj 

is  the  total  number  of  decision  package  inter¬ 
relationships 

is  the  number  of  decision  packages  for  project  i 
is  the  number  of  projects 

is  the  number  of  resource  categories  considered 
is  the  number  of  periods  in  the  planning  horizon 
is  the  amount  of  resource  type  k  required  for 
decision  package  j  of  project  i  in  time  period  p 
is  the  overall  availability  of  resource  type  k  in 
time  period  p 

r  is  the  lower  bounded  decision  package  of  project  i 

i 

I  is  an  interaction  term  representing  the  selection 

of  both  x  ,  and  x  , 
ab  cd 

Before  the  application  of  the  model  is  demonstrated,  a  discussion 
follows  of  each  constraint  and  how  the  self-interaction  matrix  concept 
can  be  used  to  determine  the  interaction  coefficient  for  the  objective 
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function. 


Resource  Constraints 


The  funding  requirements  for  each  decision  package  sub¬ 
mitted  to  the  DWMS  panel  are  for  a  period  of  five  years.  Rather 
than  considering  only  the  project  costs  for  the  first  year,  all  five 
years  can  be  considered.  In  order  to  aid  in  establishing  the  multi¬ 
year  cost  constraints,  the  cost  coefficients  for  each  DP  can  be  organ¬ 
ized  in  a  tableau  such  as  illustrated  in  Figure  9.  The  projected 
yearly  budget  can  be  estimated  as  the  sum  of  the  decision  package 
three  funding  requirements  for  all  approved  programs  for  each  year, 
or  for  the  first  year  only,  the  budget  figure  as  directed  by  AFSC . 
Mathematically,  this  can  be  expressed  as  follows: 


n  p  =  1,2, 3,4, 5 

■^T  ”  “  ai3 

p  i=l  r  The  budget  is  resource  type  1. 

Manpower,  facilities,  or  any  other  constraint  that  the  DWMS  panel 
feels  appropriate  can  be  formulated  in  a  similar  manner.  However, 
during  the  interviews  with  the  DWMS  panel  chairman  and  members, 
they  indicated  that  manpower  and  facilities  were  not  a  constraint  in 


last  year's  decision  package  evaluation  and  ranking  process. 


Mutually  Exclusive  Decision  Packages 
for  the  Same  Program 

The  mutually  exclusive  decision  package  constraint  does  not 
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Resource  Type  k  Required  Per  Year 


esource  Requirement 


3 


allow  two  decision  packages  for  the  same  program  element  to  be  in¬ 
cluded  in  the  final  portfolio.  One  constraint  is  required  for  each  pro¬ 
gram  included  in  the  DWMS  mission  area.  If  there  are  two  programs 
under  consideration  and  program  one  has  four  decision  packages  and 
program  two  has  five  decision  packages ,  then  two  constraints  are 
required. 

4 

j^lj  S  lorx11+x12+x13+Xl4S  1 

and 

5 

j?lx2j  ~  1  or  X21  +  *22  +  *23  +  x24  +  x25  3  * 

Mutually  Dependent  Decision  Packages 
for  Separate  Programs 


The  one-way  dependency  occurs  if,  before  one  program's 
decision  package  can  be  funded,  another  program's  decision  package 
has  to  be  funded.  Possibly  decision  package  two  of  program  one  is 
dependent  upon  results  obtained  from  a  specific  project  which  is  not 
included  in  program  two  until  decision  package  three  is  selected.  In 
this  particular  case,  it  should  not  be  possible  to  select  program  one, 
decision  package  two  or  higher  unless  at  least  decision  package 

three  of  program  two  is  selected.  Mathematically, 

m  i  m£ 

X  Xi:  -  2  x?s  35  0 
j=2  3  j=3  Zj 


or 


xi2  +  Xj3  +  . . .  +  xlmi  -  x23  -  ...  -  x2mz  S  0. 
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Two-way  dependency  is  similar  to  one-way  dependency 
except  the  reverse  is  also  true.  Lf  at  least  a  certain  decision  package 
for  one  program  is  chosen,  at  least  a  certain  decision  package  for 
another  program  has  to  be  selected  and  visa  versa.  This  could  occur 
where  a  high  cost  piece  of  equipment  is  required  by  two  programs 
and  the  selection  of  only  one  program  does  not  justify  the  cost.  Math¬ 
ematically, 

mi  m2 

r  xi:  -  r  x?.  =  o 

j  =  2  ^  j  =  3 

or 

x12  +  •  «  •  +  ximi  '  x23  "  •  •  •  *  x2m2  =  °* 

Compulsory  Project  with  Lower  Bounded 
Decision  Package 

This  is  the  case  where  at  least  a  certain  decision  package  for 
a  program  is  required  to  be  funded.  Perhaps  higher  management  has 
directed  that  at  least  a  certain  DP  has  to  be  funded,  or  perhaps  there 
exists  a  "letter  of  agreement"  which  states  that  the  AF  will  provide 
at  least  a  minimum  level  of  funds  for  a  joint  project.  In  last  year's 
DWMS  mission  area,  ASD  overhead,  ASD  telecommunications,  and 
4950th  Test  Wing  overhead  were  evaluated  as  compulsory  projects 
bounded  by  DP3.  At  least  DP3  had  to  be  selected,  but  if  the  DWMS 
panel  had  determined  that  DP4  was  more  valuable  to  the  AF  than 
other  possible  programs,  then  DP4  instead  of  DP3  could  be  included 
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in  the  final  portfolio.  Mathematically, 
ml 

r  x  =  i 

j  =  3  lJ 
or 


Alternate  Compulsory  Projects  with  Lower 
Sounded  Decision  Packages 

This  is  the  case  where  higher  management  has  said  that 
either  at  least  decision  package  two  of  program  one  or  at  least  deci¬ 
sion  package  two  of  program  two  has  to  be  funded  but  it  does  not 
matter  which  one.  The  0-1  IP  model  is  allowed  to  select  the  decision 
package  which  contributes  the  most  to  the  project  portfolio  so  as  to 

maximize  the  benefits  to  the  organization.  Mathematically, 

m  i  m2 

T  x,<  +  T  x2-  *  1 
j=2  V  j  =2 


x12  +  •  •  •  +  xim1  +  x22  +  *  ’  '  +  x2mj  ~ 

Note,  that  if  x22  is  selected  first,  there  is  no  longer  a  constraint  on 
Xj.  so  that  Xj  j  can  be  included  in  the  final  portfolio. 


Project  Interactions 


After  the  DWMS  panel  has  completed  the  self- interaction 
matrix  for  all  programs  under  consideration,  the  information  con¬ 
tained  in  the  matrix  can  be  used  to  obtain  the  interaction  terms  of 


1 
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the  objective  function.  The  procedure  is  a  two  step  process.  First, 
for  every  nonzero  element  in  the  interaction  matrix,  which  corres¬ 
ponds  to  an  interaction  between  project  "a"  decision  package  "b"  and 
project  "c"  decis ion  package  "d,"  a  new  variable,  lg,  is  created. 
Second,  the  value  of  that  nonzero  element,  which  represents  the 
incremental  benefit  change  caused  by  selecting  project  "a"  decision 
package  "b"  and  project  "c"  decision  package  "d,"  is  the  coefficient, 
bg ,  for  use  in  the  objective  function. 

Since  all  variables  are  0-1  and  I  can  only  equal  one  if  both 
and  xc(j  are  equal  to  one  because  Ig  =  xabxcd>  the  following  two 
constraints  are  required  for  each  interaction: 

Kab  +  Xcd'lsSl  (i) 

and 

-  Xab  -  xcd  +  2Is  S  0  <Z) 

To  demonstrate  that  these  two  constraints  cause  lg  to  have  the  appro¬ 
priate  values  consider  the  following: 

1.  When  x^  =  xc^  =  0,  constraint  1  does  not  constrain  Ig, 
but  constraint  2  causes  Ig  =  0. 

2.  When  x ^  =0,  xc£j  =  1  or  xa^  =  1,  xccj  =  0,  constraint  1 
and  constraint  2  cause  Ig  =  0 . 

3.  Finally,  when  xa^  =  x  ^  =  1,  constraint  1  becomes 

1  +  1  -  I  5  1  or  15  1 
s  s 


1 

1 


I 

I 


I 
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and  constraint  2  becomes 


-  1  -  1  +  2IS  5  0  or  Is  S  1 

and  both  can  only  be  satisfied  if  Ig  =  1  (McMillan,  1975:500). 

Example  l--R&tD  Project  Selection  with 
Project  Interrelationships --Method  1 

A  short  example  will  demonstrate  how  project  interrelation¬ 
ships  can  be  included  in  the  R&D  project  selection  model  when  the 
project  interrelationships  are  expressed  as  incremental  benefit 
changes.  Assume  that  there  are  two  projects  each  having  two  deci¬ 
sion  packages  under  consideration.  The  funding  requirements  for 
each  decision  package  and  all  other  appropriate  decision  package 
information  has  been  submitted  to  a  panel  of  experts  which  has  to 
evaluate  and  select  the  best  project  portfolio  within  given  constraints. 
The  panel  proceeds  as  follows: 

1.  Evaluate  the  benefits  of  the  separate  decision  packages 
to  the  organization. 


Program 

Decision  Package 

Cost 

Benefit 

1 

1 

10 

10 

2 

20 

15 

2 

1 

8 

6 

2 

19 

14 
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2.  Construct  the  self- interaction  matrix  and  eliminate  non- 


feasible  interactions. 


3.  For  all  feasible  interactions,  evaluate  whether  selecting 
the  corresponding  pair  of  decision  packages  has  an  additional  impact 
on  the  benefits  to  the  organization.  For  no  impact,  enter  zero.  For 
any  impact,  enter  the  magnitude  of  that  impact.  For  this  example, 
all  decision  packages  interact.  Note  that  Xjj  and  x2  ^  have  a  2  unit 
cannibalistic  effect  on  the  organization's  benefits.  The  other  inter¬ 
actions  are  shown  below. 


4.  For  each  element  that  has  a  non-zero  value,  create  a 
new  variable  Ig.  A  new  self- interaction  matrix  can  be  used  to  aid  in 


the  necessary  bookkeeping. 


h 

22' 

li 

X3 

-- 

12  X 

n 

\ 

The  variable  1^  corresponds  to  the  selection  of  both  and 


X2  r 

5.  Use  this  information  plus  a  budget  constraint  of  30  to 
formulate  a  0-1  EP  model  to  solve  this  problem. 

Maximize:  z  =  lOx^  +  15xj2  +  6x31  +  *^x22 

-  2Ij  -  3I2  +  3I3  +  I4 

Subject  to: 

(1)  Resource  Constraint 


lOx^  +  20x^  +  8x21  +  ^x22  ~  ^ 

(2)  Mutually  Exclusive  Decision  Packages 


X11  +  x12  S  1 


X21+X22S1 


(3)  Project  Interaction  Constraints 


Xll+X21 


-Xll-X21+2Il-° 


X11  +X22  '  X2  5  1 


'  X11  -  x22  +  2I2 
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x12  +  X21  '  X3  -  1 
x12  *  X21  +  2I3  "° 


X12  +  X22  -  h  *  1 


X11  -  x22  +  2I4  -° 


(4)  0-1  Constraints 

All  variables  =0,1 

The  solution  to  this  problem  is  to  select  projects  xjj  and 
The  cost  to  the  organization  is  28,  and  the  selected  portfolio 
provides  the  maximum  possible  benefits  which  is  24. 


Example  2--R&iD  Project  Selection  with 
Project  Interrelationships--Method  2 

An  alternate  0-1  IP  formulation  can  be  used  when  the  total 
benefit  for  selecting  a  particular  project  combination  is  known.  This 
is  the  case  where  it  is  easier  to  determine  the  total  benefits  for 
selecting  two  projects  rather  than  determining  the  incremental  change 
in  benefits.  The  previous  example  will  be  used  to  demonstrate  this 
procedure . 

1.  Evaluate  the  benefits  of  the  separate  decision  packages 
to  the  organization. 
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-t  '  * 


Program 

Decision  Package 

Cost 

Benefit 

1 

1 

10 

10 

2 

20 

15 

2 

1 

8 

6 

2 

19 

14 

2.  Construct  the  self-interaction  matrix  and  eliminate  non- 
feasible  interactions. 


3.  For  all  feasible  interactions,  determine  the  total  benefits 
to  the  organization  for  selecting  the  corresponding  pair  of  decision 
packages.  Compare  the  benefits  with  interactions  to  the  summed 
total  of  benefits  from  the  two  individual  decision  packages .  If  there 
is  no  difference,  enter  zero  in  that  element  of  the  matrix.  If  a  dif¬ 
ference  exists,  enter  the  interaction  benefits. 


4.  For  each  element  that  has  a  non-zero  value,  create  a 

new  variable  I  . 

s 
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x 


iT\ 


21 


12 


11 


The  selection  of  1^  in  the  final  solution  implies  that  projects 

x,  ,  and  x  are  selected. 

11  21 

5.  For  each  Ig,  determine  the  resource  level  required  for 
this  particular  decision  package  combination.  Again,  the  concept  of 
a  self- interaction  matrix  will  be  used.  Under  the  assumption  of  no 
overlap  in  project  resource  utilization,  the  resource  requirement  for 
Ig  would  be  the  summed  total  of  the  resource  requirements  for  xa^ 
and  xccj.  If  there  was  an  interdependency  due  to  overlap  in  project 
resource  requirements,  then  the  actual  resource  requirement  would 
be  entered  into  the  matrix.  In  this  example,  there  is  no  interdepend¬ 
ency  in  funds  required  so  the  funds  required  by  Ig  is  the  summed 
total  of  funds  required  for  xa^  and  xccj. 


29 

39 

1  — S 
22 

18 

28 

IT' 

12  X 

11 

6.  Formulate  a  0-1  IP  model  with  a  budget  constraint  of  30. 
Maximize:  z  =  lOx^j  +  15Xj2  +  ^x2 1  +  ^x22 

+  14I:  21I2  +  24I3  +  30i4 
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Subject  to: 

(1)  Resource  Constraint 

10x11  +  20x12  +  8x2i  +  ^x22 
+  181  j  +  29I2  +  28I3  +  39I4  5  30 

(2)  Mutually  Exclusive  Decision  Packages 

x,  ,  +  x  SI 
11  12 

x2i+x22S  1 

(3)  Project  Interaction  Constraints 

Under  this  approach,  project  interaction  constraints 
are  changed  to  the  following  form, 

xab  +  xcd  +  ls~1’ 
for  each  Ig . 

X11  +x21  +  ll  3  1 

XU  +  x22  +  h  ~  1 

x12  +  X21  +  X3  3  1 

X12  +  X22  +  X4  3  1 

(4)  0-1  Constraints 

All  variables  =0,1 

Solving  this  problem  yields  the  selection  of  I3  which  implies 

the  selection  of  projects  x  _  and  x  because  I,  =  x  x  .  Total 

ic.  Z  1  j  IZ  Z 1 

benefits  are  24  for  a  cost  of  28. 

The  first  method  of  incorporating  decision  package  inter¬ 


dependencies  is  suggested  for  use  by  the  DWMS  panel  for  two  reasons. 
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First,  the  DWMS  panel  believed  that  the  fiscal  constraint  was  the 
only  constraint  pertinent  to  their  selection  process.  Since  the  panel 


had  no  authority  to  change  any  funding  requirements  for  any  decision 
packages,  advantage  could  not  be  taken  of  the  possibility  that  funding 
two  programs  together  required  less  funds  than  funding  each  program 
separately.  Secondly,  due  to  the  large  number  of  programs  and  deci¬ 
sion  packages,  it  would  be  easier  to  evaluate  the  benefits  of  each 
decision  package  separately  and  then  evaluate  the  change  in  benefits 
caused  by  funding  particular  combinations  of  decision  packages. 

Next,  a  more  complicated  example  with  ten  programs  and 
thirty-seven  decision  packages  will  demonstrate  the  detailed  use  of 
the  model.  The  final  output  will  be  a  complete  ranking  of  decision 
packages  for  submission  to  the  next  higher  level  of  management  in 
the  ZBB  process. 


Example  3- -Multi-Program  R&D 
Project  Selection 

The  example  developed  to  demonstrate  the  full  use  of  the 
model  contains  ten  programs  and  thirty-seven  decision  packages.  It 
is  assumed  that  the  mission  area  review  panel  has  already  met  and 
determined  the  benefits  of  each  decis ion  package  to  the  AF.  The 
results  from  the  decision  package  evaluations  and  the  multi-year 
funding  requirements  for  each  decision  package  are  shown  in  Table  I. 
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AFSC  has  provided  the  panel  with  a  budget  amount  of  35  for  the  first 
/ear  of  the  five  year  funding  requirements.  For  all  other  years,  two 
through  five,  the  decision  package  threes  for  all  approved  programs 
are  summed  and  represent  the  projected  budgec  levels  for  the  appro¬ 
priate  year.  Two  programs,  two  and  five,  are  new  starts,  and  only 
decision  packages  one  and  four  for  each  are  submitted.  Program 
nine  represents  a  program  which  is  completed  at  the  end  of  year 
three  and  therefore,  requires  no  funds  in  years  four  and  five.  Pro¬ 
gram  two  is  a  program  whose  costs  are  low  in  the  first  year  but 
rapidly  escalate  in  the  next  four  years.  Program  five  represents  a 
new  start  which  requires  a  large  percentage  of  the  total  yearly  funds 
available.  Finally,  program  ten  represents  a  program  which  pro¬ 
vides  high  benefits  to  the  organization  but  requires  a  relatively  high 
investment  in  order  to  be  selected. 

The  procedures  required  to  formulate  the  problem  are  simi¬ 
lar  to  those  in  example  one. 

1.  Evaluate  the  benefits  of  the  separate  decision  packages  to 
the  organization  using  the  scoring  model  presented  in  the  beginning  of 
this  chapter.  Calculate  the  benefit-cost  ratio  for  each  decision  pack¬ 
age  based  upon  the  first  year  package  cost.  The  results  of  this  eval¬ 
uation  are  shown  in  Table  I. 

2.  Construct  the  self-interaction  matrix  and  eliminate  non- 
feasible  interactions. 
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TABLE  I 


PROGRAM  INFORMATION  FOR  EXAMPLE  3 


Program 

Decision 

Package 

Funding  Requirements  by 

12  3  4 

Year 

5 

Bene¬ 

fits 

B  enefit- 
Cost 

Ratio 

1 

1 

.7 

.8 

.9 

1.1 

1.2 

4 

5.71 

2 

1.4 

1.5 

1.6 

1.8 

1.9 

28 

20.0 

3 

2.7 

2.8 

3.0 

3.2 

3.4 

34 

12.59 

4 

3.8 

3.9 

4.0 

4.2 

4.3 

45 

11.84 

5 

5 .  1 

4.2 

4.3 

4.3 

4.3 

47 

9.22 

2 

1 

1.6 

3.6 

4.2 

4.8 

5.0 

30 

18.75 

4 

3.  1 

4.0 

4.3 

4.9 

5.2 

39 

12.58 

3 

1 

1.3 

1.4 

1.6 

1.7 

1.9 

30 

23.08 

2 

2.6 

2.9 

3.  1 

3.5 

3.8 

42 

16.  15 

3 

5.2 

5.7 

6.3 

6.9 

7.6 

50 

9.62 

4 

6.1 

6.7 

7.4 

8.1 

8.9 

51 

8.36 

4 

1 

.6 

.7 

.8 

1.0 

1.1 

12 

20.0 

2 

1.1 

1.2 

1.3 

1.5 

1.7 

22 

20.0 

3 

2.2 

2.4 

2.2 

2.9 

3.2 

29 

13.  18 

4 

3.4 

3.6 

3.8 

4.0 

4.  1 

30 

8.82 

5 

1 

10.2 

10.8 

11.5 

12.  1 

12.9 

11 

1.08 

4 

12.4 

12.9 

13.4 

13.9 

14.2 

15 

1.21 

6 

1 

1.  1 

1.2 

1.3 

1.3 

.8 

19 

17.27 

2 

2.1 

2.3 

2.4 

2.3 

1.8 

25 

11.90 

3 

4.2 

4.5 

4.9 

4.0 

2.6 

28 

6.67 

4 

5.1 

5.4 

5.7 

4.9 

2.9 

30 

5.88 

7 

1 

.9 

1.0 

1.1 

1.4 

1.6 

13 

14.44 

2 

1.9 

2.  1 

2.3 

2.4 

2.6 

18 

20.0 

3 

3.7 

4.  1 

4.3 

4.6 

4.9 

26 

7.03 

4 

4.8 

5.3 

5.6 

6.0 

6.3 

28 

5.83 
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TABLE  I  (Cont. ) 


Program 

Funding  Requirements  by 

Decision 

Package  1234 

Year 

5 

Bene¬ 

fits 

Benefit- 

Cost 

Ratio 

8 

1 

.5 

.7 

.8 

.9 

1.2 

16 

32.0 

2 

1.1 

1.2 

1.3 

1.5 

1.7 

24 

21.82 

3 

2.2 

2.3 

2.3 

2.4 

2.5 

29 

13.  18 

4 

3.9 

3.9 

4.0 

4.0 

4.  1 

30 

7.69 

9 

1 

1.3 

1.0 

.7 

0 

0 

31 

23.85 

2 

2.  1 

1.8 

.8 

0 

0 

36 

17.  14 

3 

3.6 

3.4 

1.2 

0 

0 

41 

11.  39 

4 

4.9 

4.2 

2.6 

0 

0 

44 

8.98 

10 

1 

6.4 

6.7 

7.  1 

7.4 

7.8 

27 

4.22 

2 

7.9 

8.2 

8.5 

8.8 

8.9 

40 

5.06 

3 

8.8 

9.  1 

9.3 

9.6 

9.9 

47 

5.34 

4 

10. 1 

10.2 

10.3 

10. 1 

10.0 

49 

4.85 

Estimated 

Funds 

Available 

n 

£  M31p 
i=l 

35 

(AFSC 

Figure) 

34.3 

34.0 

33.6 

34.  1 

p=2,3,4,5 
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i 

i 

i 

For  a  large  problem  with  many  possible  interactions,  it 
would  be  easier  to  construct  the  self- interaction  matrix  only  for  those 
projects  that  interact.  Figure  10  shows  a  portion  of  the  total  self¬ 
interaction  matrix  containing  programs  one,  two,  four,  and  eight. 

3.  For  all  feasible  interactions,  evaluate  whether  selecting 
the  corresponding  pair  of  decision  packages  has  an  additional  impact 
on  the  benefits  to  the  organization. 

For  this  example,  it  is  assumed  that  programs  are  not 
interdependent;  therefore,  all  the  elements  in  the  self- interaction 
matrix  are  zero.  Selecting  two  projects  does  not  have  either  a 
synergistic  or  cannibalistic  effect  on  the  total  benefits  from  the  pro¬ 
ject  portfolio. 

4.  Flag  compulsory  decision  packages .  At  least  decision 
package  33  must  be  chosen.  Either  at  least  decision  package  42  or 
at  least  decision  package  62  must  be  chosen. 

5.  The  problem  is  now  formulated  using  the  0-1  IP  model. 

The  computer  formulation  is  shown  in  Appendix  B. 

The  resulting  0-1  IP  problem  was  solved  by  the  use  of  Multi 
Purpose  Optimization  System  (MPOS)  on  a  Cyber  74  computer  at 
ASD.  The  advantages  of  MPOS  are  that  the  language  allows  the  user 
to  state  the  problem  in  English  and  algebraic  notation  and  the  effi¬ 
ciency  of  the  MPOS  0-1  algorithm  (Cohen  and  Stein,  1978:1). 

In  order  to  obtain  the  rank  order  of  decision  packages  as 
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Figure  10:  Portion  of  Total  Self-Interaction  Matrix 


i 
) 

specified  by  ZBB,  the  program  was  first  run  looking  only  at  the 
funding  requirements  for  the  first  year.  In  this  case,  the  budget 
constraint  of  35,  which  was  provided  by  AFSC,  was  used  and  the 
resulting  portfolio  is  shown  in  Table  II.  Next,  the  problem  was  run 
considering  the  funding  requirements  for  all  five  years  in  the  FYDP. 

Again,  the  resulting  portfolio  is  shown  in  Table  II.  The  multi-year 
nature  of  the  problem  will  change  the  portfolio  if  any  year's  budget 
constraint  other  than  the  first  year's  constraint  becomes  binding.  In 
this  example,  the  portfolio  did  change  with  decision  package  15  being 
selected  in  place  of  decision  packages  73  and  93.  It  would  be  a 
management  decision  which  portfolio  to  select. 

In  order  to  determine  how  sensitive  the  project  portfolio 
was  to  changes  in  the  funding  level  and  to  aid  in  the  ranking  process, 
the  program  was  run  considering  only  the  first  year  and  varying  the 
budget  constraint  from  a  low  value  of  10  to  a  high  value  of  55.  All 
programs  would  be  selected  if  58.8  funds  were  available.  The 
results  for  an  incremental  change  of  five  are  shown  in  Table  II  and 
are  the  optimal  portfolio  at  that  funding  level.  Smaller  incremental 
changes  could  be  performed  around  the  budget  figure  if  additional 
analysis  was  necessary. 

One  of  the  most  difficult  tasks  in  ZBB  is  the  actual  ranking 
of  decision  packages  which  is  the  ultimate  end  result  of  the  process. 

Cheek  claims  that  what  is  required  is  an  efficient  method  which 
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focuses  and  directs  the  organization’s  resources  towards  those  pro¬ 
grams  which  provide  the  most  promise  of  achieving  the  preagreed 
needs  of  the  organization  (Cheek,  1977:54) .  The  1982-1986  POM 
guidance  provided  little  detail  on  the  ranking  procedures  to  be  used, 
leaving  the  decision  up  to  each  Mission  Area  Review  panel.  The  only 
guidance  was  that  the  ranking  should  normally  be  accomplished  in 
conjunction  with  mission  area  analysis  and  the  AF  Planning  Guide 
(ASD/ACB,  1979:57). 

The  following  procedure  will  be  suggested  for  use  by  the 
DWMS  panel  in  ranking  decision  packages  and  will  be  demonstrated 
by  this  example. 

1.  All  compulsory  projects  and  the  alternate  compulsory 
projects  that  have  been  selected  are  ranked  at  the  top  of  the  list  of 
decision  packages .  For  this  example,  x-j^  is  a  compulsory  package, 
and  X42  is  the  first  alternate  compulsory  project  to  be  selected. 

Both  appear  at  the  top  of  the  ranking  list. 

2.  Sensitivity  analysis  is  performed  on  the  budget  constraint 
around  the  estimated  budget  figure  for  the  first  year  of  the  FYDP. 

The  range  is  predetermined  by  the  DWMS  panel  and  is  sufficient  to 
clearly  differentiate  between  those  projects  which  will  definitely  be 
selected,  marginal  projects,  and  those  which  will  not  definitely  be 
selected.  The  budget  constraint  will  be  incremented  around  the 

estimated  budget  figure  in  order  to  determine  which  of  the  marginal 
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programs  should  be  selected  and  the  order  of  selection.  A  tradeoff 
decision  has  to  be  made  when  determining  the  appropriate  increment 
size  between  the  number  of  computer  runs  required  and  the  amount 
of  detail  desired  about  the  selection  order  for  marginal  programs. 

If  the  budget  figure  for  the  first  year  of  the  FYDP  for  the  DWMS  mis¬ 
sion  area  is  380,  a  suggestion  would  be  to  increment  the  budget  con¬ 
straint  from  340  to  420  in  increments  of  10. 

For  this  example,  the  budget  for  the  first  year  is  35.  A 
reasonable  range  over  which  to  increment  the  budget  constraint  is 
from  25  to  45  in  increments  of  5.  Table  III  illustrates  the  portfolio 
for  a  budget  constraint  of  25  and  the  additional  decision  packages  that 
are  selected  as  the  budget  constraint  is  increased. 

3.  Rank  all  decision  packages  that  are  included  in  the  port¬ 
folio  for  the  lowest  budget  constraint  according  to  their  benefit-cost 
ratio.  In  this  example,  all  decision  packages  that  are  included  in 
the  portfolio  for  a  resource  level  of  25  except  for  compulsory  pro¬ 
jects  are  ranked  according  to  their  benefit-cost  ratio. 

4.  As  additional  decision  packages  are  added  at  higher 
resource  levels,  order  the  added  decision  packages  according  to 
their  benefit-cost  ratio  without  changing  previously  ranked  decision 
packages.  For  this  example,  when  the  resource  level  is  increased 
to  30  from  25 ,  decision  pac  cages  72,  92,  43,  62,  and  83  are  now 
included  in  the  desired  portfolio.  These  additional  decision  packages 
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are  ranked  according  to  their  benefit-cost  ratio.  Tht-  ranking  for 


a  resource  level  of  25  is  not  changed. 

5.  Repeat  step  four  until  the  upper  limit  of  the  predeter¬ 
mined  resource  range  has  been  obtained. 

6.  All  remaining  decision  packages  are  ranked  according 
to  their  benefit-cost  ratios. 

For  this  example,  the  final  decision  package  ranking  is 
shown  in  Table  III.  If  available  funds  are  reduced  below  35,  DP73 
would  be  the  first  to  be  eliminated.  DP  15  would  be  the  first  decision 
package  added  to  the  portfolio  if  the  funds  available  are  increased 
above  35. 


Summary 

This  chapter  presented  a  scoring  model  that  the  DWMS  panel 
can  use  to  evaluate  the  benefits  of  each  decision  package  and  a  pro¬ 
ject  selection  model  which  selects  the  decision  package  portfolio  with 
the  maximum  benefits  to  the  AF.  The  next  chapter  presents  the 
conclusions  drawn  from  this  study  and  the  final  recommendations  to 
the  DWMS  panel  for  improving  the  1983-1987  POM  process. 


TABLE  III 


FINAL  DECISION  PACKAGE  RANKING 


Decision 

Package 


Resource 

Levels 


Decision 

Package 


Resource 

Levels 


Chapter  6 


CONCLUSIONS  AND  RECOMMENDATIONS 

This  chapter  presents  the  conclusions  drawn  from  this 
research  effort,  and  based  upon  those  conclusions,  recommendations 
to  the  DWMS  Mission  Area  Review  panel  for  improving  the  1983-1987 
POM  process.  The  recommendations  will  be  integrated  with  improve¬ 
ments  that  already  have  been  suggested  by  the  corporate  management 
at  ASD. 

Several  key  conclusions  drawn  from  this  study  are  as  follows: 

1)  the  decision  package  information  collected  from  the  pro¬ 
gram  managers  is  not  cons istently  prepared; 

21  it  is  very  difficult  for  the  DWMS  panel  members  to  eval¬ 
uate  the  decision  packages  with  respect  to  AF  long-range  goals 
because  the  objectives  of  each  decision  package  and  the  goals  of  the 
AF  are  not  explicitly  stated; 

3)  the  internal  and  external  consistency  of  the  DWMS  deci¬ 
sion  package  evaluation  process  needs  to  be  improved  so  that  all 
panel  members  can  evaluate  each  decision  package  equivalently; 

4)  an  appropriate  selection  model,  which  allows  for  multiple 
constraints  and  project  interdependencies,  needs  to  be  developed;  and 
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5)  finally,  a  timetable  of  events  needs  to  be  developed  in 
order  to  permit  a  thorough  evaluation  of  all  DWMS  decision  packages 
to  be  performed. 

In  order  to  increase  the  consistency  of  the  decision  packages 
and  to  tie  those  decision  packages  to  AF  long-range  goals,  ASD/AC 
(Comptroller)  and  ASD  corporate  management  should  provide  further 
preparation  guidance  to  the  program  managers  and  the  Mission  Area 
Review  panel  members.  This  guidance  could  be  in  the  form  of  either 
a  briefing  or  detailed  letter  which  would  discuss  the  following: 

1)  the  purpose  and  philosophy  of  the  POM  ranking  process, 

2)  the  type  of  information  which  should  be  included  in  the 
decision  packages, 

3)  the  importance  of  providing  accurate  estimates, 

4)  how  vital  it  is  to  the  AF  that  the  decision  package  goals 
be  related  to,  as  clearly  as  possible,  AF  goals,  and 

5)  a  clear  definition  of  exactly  what  those  AF  goals  are. 

If  well  prepared  decision  packages  are  received  by  the  Mission  Area 
Review  panels,  especially  the  DWMS  panel,  the  process  of  selecting 
the  most  appropriate  decision  packages  could  be  enhanced  consider¬ 
ably.  It  would  be  less  probable  that  an  inappropriate  decision  package 
would  be  selected. 

To  further  increase  the  consistency  of  the  decision  packages, 
ASD/AC  should  review  the  packages  before  they  are  sent  to  the 


Mission  Area  Review  panels.  If  a  package  did  not  meet  the  quality 
control  standards,  then  it  would  be  sent  back  to  be  reworked  by  the 
program  manager. 

In  order  to  improve  the  consistency  of  evaluating  decision 
packages  by  the  DWMS  panel,  it  is  recommended  that  the  scoring 
model  developed  in  this  thesis  be  used  to  measure  the  benefits  to  the 
AF  of  each  decision  package.  Once  the  benefits  have  been  deter¬ 
mined,  the  0-1  integer  programming  model  should  be  used  to  develop 
the  decision  package  ranking  for  the  POM. 

In  order  to  aid  the  DWMS  panel  chairmen  and  ASD/XR  in 
scheduling  events  for  the  1983-  1987  POM  evaluation  process ,  the 
following  timetable  has  been  developed.  This  timetable  incorporates 
improvements  to  the  ASD  POM  process  suggested  by  ASD's  corporate 
management  (ASD/AV,  1980: 15- 16)  and  proposed  by  this  research 
effort.  At  the  beginning  of  October,  the  Mission  Area  Review  panels 
members  will  be  selected  and  along  with  the  program  managers 
attend  a  briefing  conducted  by  ASD/AC  and  ASD  corporate  manage¬ 
ment.  This  briefing  will  cover  ASD's  strategic  plan,  AF  goals,  the 
philosophy  of  ZBB,  and  decision  package  preparation.  After  this 
briefing,  the  program  managers  will  have  two  weeks  to  prepare  all 
decision  packages  and  submit  them  back  to  ASD/AC  for  quality  con¬ 
trol.  During  the  third  week  of  October,  ASD/AC  will  review  all  the 
decision  packages  to  determine  if  they  are  consistently  prepared. 


Those  decision  packages  that  do  not  meet  the  quality  control  stand¬ 
ards  will  be  returned  to  the  program  managers  for  rework.  These 
three  weeks  will  allow  each  Mission  Area  Review  panel  to  review  the 
most  current  BES  which  was  submitted  in  September,  the  Air  Force 
Planning  Guide,  and  the  "P"  series  documents  to  determine  the  AF 
needs  in  each  mission  area.  Also,  during  these  three  weeks,  each 
panel  can  discuss  and  determine  the  decision  package  evaluation  and 
selection  strategy  that  will  be  used  to  rank  and  select  decision  pack¬ 
ages.  This  research  has  suggested  and  demonstrated  one  such 
strategy. 

Around  the  last  of  October,  the  Mission  Area  Review  panels 
receive  their  appropriate  decision  packages  and  the  individual  review 
process  begins.  Following  the  Vanguard  briefings  in  November, 
each  panel  will  initially  rank  their  decision  packages  based  on  mission 
requirements.  The  scoring  and  decision  package  selection  models 
discussed  in  this  thesis  have  been  developed  for  use  by  the  DWMS 
Mission  Area  Review  panel.  The  results  from  these  initial  rankings 
are  presented  to  the  Program  Review  Group  for  review  in  late  Novem¬ 
ber.  When  the  final  fiscal  guidance  is  received  in  early  December, 
the  rankings  will  be  adjusted  as  necessary,  and  the  results  presented 
to  the  Decision  Review  Group  for  review  and  approval. 

Areas  for  further  research  include  the  following: 

1)  an  evaluation  of  the  cost  effectiveness  of  starting  the 

154 


decision  package  rankings  at  the  Product  Division  level  rather  than 
starting  at  either  AFSC  or  the  Air  Staff; 

2)  an  analysis  of  the  different  methods  that  firms  use  to 
evaluate  R&cD  projects; 

3)  expand  the  presented  0-1  integer  programming  model  to 
include  multiple  criteria;  and 

4)  finally,  an  investigation  into  the  best  method  to  rank 
decision  packages  for  the  POM  when  using  the  0-1  integer  program¬ 
ming  R&D  project  selection  model  presented  in  this  thesis. 
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APPENDIX  A 

SCORING  MODEL  FOR  USE  IN  EVALUATING 
DWMS  DECISION  PACKAGES 
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A  description  of  the  scoring  model  suggested  for  use  by  the 
DWMS  Mission  Area  Review  panel  follows.  The  individual  score  for 
program  i  decision  package  j  from  panel  member  z  is  given  by: 
[individual  Score]^ z  =  ^[Military  Value  Score]..  +  [Feas ibility  Score]- 


+  [Resource  Acceptability  Score]- 


1 


(1) 

The  total  score  or  objective  function  coefficient  for  program  i  deci¬ 
sion  package  j  is  determined  by: 

f 

b..  =  [Total  Score] ••  =  [Group  Importance  Score]-,  x  E  [individual 

Z=1 


Score] 


ijz 


i—  1 i 2 |  ...I  n 
J  -  1  »  2 ,  » »  .  ,  ^  ^ 


where  f  is  the  total  number  of  members  on  the  DWMS  Mission  Area 
Review  panel,  n  is  the  number  of  projects,  and  m^  is  the  number  of 
decision  packages  for  project  i. 

A  checklist  is  now  presented  which  will  aid  the  DWMS  panel 
members  in  evaluating  each  decision  package  against  the  provided 
criteria.  For  each  evaluation  criteria,  a  list  of  typical  questions 
that  should  be  considered  is  provided.  After  reviewing  the  provided 
list  and  any  other  questions  that  the  DWMS  panel  deems  relevant, 
each  decision  package  is  evaluated,  and  a  number  representing  the 
value  of  that  decision  package  to  the  AF  selected  from  the  provided 
scale.  Once  each  decision  package  has  been  evaluated  against  the 
criteria,  a  decision  package  individual  score  is  obtained  by  using 
formula  1. 
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Scoring  Model  Checklist 


I.  Military  Value 
(Relevance) 


1 

L_ 


Low 

(No) 


High 

(Yes) 


A.  Mission  Accomplishment 

1.  If  the  program  is  technically  successful,  will  the 
resulting  system  satisfy  the  need? 

2.  What  is  the  final  performance  of  the  system  being  pro¬ 
posed  by  this  DP? 

3.  Will  the  proposed  system  be  effective? 

B.  Survivability 

1.  Will  this  DP  result  in  a  system  that  is  survivable  in  the 
expected  threat  environment? 

C.  Long-Range  Goals 

1.  Is  this  DP  compatible  with  current  AF  strategy? 

2.  Are  the  objectives  of  this  DP  related  to  AF  objectives? 

D.  System  Improvement 

1.  Does  this  DP  contribute  to  the  improvement  of  a  current 
or  future  system? 

E.  Timing 

1.  How  soon  is  this  system  needed? 
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2.  What  is  the  expected  development  time? 


3.  What  is  the  probability  of  obsolescence?  Will  the 
system  be  obsolete  before  it  is  developed?  Will  the  threat  change 
s ignificantly  so  that  this  program  is  not  needed? 

II.  Feasibility 

12  3  4 

I - 1 - 1 - 1 

Low  High 

A.  Once  this  program  is  started,  will  it  be  successfully  com¬ 
pleted? 

B.  How  realistic  is  this  DP  in  the  current  political,  economic, 
and  technical  environment? 

1.  Technical  Feasibility 

a.  What  is  the  technical  risk  of  this  DP? 

b.  Is  the  required  technology  available  or  does  the  DP 
depend  upon  a  laboratory  research  program?  Will  the  technology 
from  the  laboratory  be  available  when  required? 

c.  Have  alternative  technical  solutions  been  explored 
for  the  relevant  technological  bottlenecks? 

d.  Are  the  required  components  available  but  have  not 
been  integrated  to  determine  how  they  perform  together? 

e.  Does  this  program  rely  on  "off-the-shelf"  com¬ 


ponents  ? 
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2.  Political  Feasibility 


a.  Who  is  the  program's  champion? 

b.  Does  this  DP,  if  selected  by  this  panel,  have  a  high 
probability  of  being  funded? 

3.  Organizational  Feas ibility 

a.  Is  the  organization  capable  of  performing  the  work0 

b.  Are  the  required  engineers,  scientists,  material, 
facilities,  and  management  available? 

c.  How  well  thought  out  is  the  program?  Does  the 
program  sound  realistic? 

III.  Resource  Acceptability 

12  3  4 

Low  ^  ^  High 

A.  General 

1.  Given  that  this  program  is  desired,  is  the  organization 
capable  of  performing  the  work? 

2.  Does  this  DP  require  a  large  percentage  of  the  available 
resources  ? 

3.  Is  this  program  efficient?  Is  this  particular  DP  the 
most  efficient  method  to  achieve  the  program’s  objectives?  Is  the 
job  being  done  for  the  least  cost? 

4.  Are  the  future  costs  realistic? 
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B.  Timing 


1.  Will  resources  be  available  when  required0 

2.  Are  the  development  costs  versus  time  realistic? 

IV.  Importance 

12  3  4 

I _ I _ I _ 1 

Low  High 

A.  How  important  is  this  DP  to  AF  strategy? 

B.  How  important  is  this  DP  with  respect  to  all  other  decision 
packages  in  the  DWMS  mission  area? 

C.  How  does  this  DP  relate  to  the  following  priority  structure? 

1.  Priority  1  -  DP  is  directed  toward,  or  is  part  of,  a 
broader  effort  aimed  at  meeting  a  must  requirement  to  achieve  an 
AF  objective  or  goal. 

2.  Priority  2  -  DP  is  aimed  at  a  less  favored  approach 
among  several  to  meeting  a  must  requirement. 

3.  Priority  3  -  Project  only  contributes  to  a  desired  but 
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not  required  goal. 


Compulsory  Projects 

Next,  the  DWMS  panel  must  flag  compulsory  projects .  Com¬ 
pulsory  projects  will  be  formulated  as  constraints. 

1.  Is  at  least  a  certain  DP  required  by  law  or  regulation0 

2.  Has  the  AF  committed  itself  to  provide  a  certain  level 
of  funds  to  a  joint  program? 

3.  Does  it  make  sense  to  fund  an  overhead  program  element 
at  some  other  minimum  level  than  DP  1? 

Interdependencies 

The  following  checklist  will  aid  in  determining  the  benefit 
adjustments  required  by  considering  interdependent  programs. 

1.  Does  this  DP  contribute  to  the  standardization  with  other 
programs  ? 

2.  What  are  the  effects  on  other  R&D  programs? 

3.  Is  there  an  overlap  in  resource  utilization  between  pro¬ 
grams?  Can  two  programs  share  manpower,  facilities,  or  funds0 
Can  two  programs  be  completed  together  for  less  cost  than  each 
individually0 

4.  Can  the  technological  breakthroughs  on  one  program 
affect  the  outcome  of  another  program? 

5.  Does  selecting  two  programs  have  a  synergistic  or 
cannibalistic  effect  upon  each  other?  What  is  the  magnitude  of  that 
effect  ? 
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TITLE 

EXAMPLE  GF  3-1  IF  APPLIED  TO  THE  PCM  R*D  PROGRAM  SELECTION  PROBLEM 
DSZ1IP 

« 

*  OBJECTIVE  FUNCTION  -  MAXIMIZE  PROGRAM  BENEFITS 


MAXIMIZE 

4X 1 J+2SX12+34X 13+45K 1 4+47X 15  +33X21+39X24  +35X31+42X32t53X33+5!X34 
+1ZX41+2ZX4Z+29X43+33K44  +11X51+15X54  +19X41+25X62+23143+33X44 
+16X31+24X8Z+29IB3+33X84 
+27X101+48X132+47X103+49X194 


+13X71+18X72+26X73+28X74 

+31X91+36X92+41X93+44X94 


+  CONSTRAINTS 
CONSTRAINTS 


i 

+  MUTUALLY  EXCLUSIVE  PROGRAM  VERSIONS 

+  ONLY  ONE  DECISION  PACKAGE  FOR  EACH  PROGRAM  ELEMENT  CAN  BE 
+  INCLUDED  IN  THE  FINAL  PORTFOLIO 


1.  X11+X12+X13+X14+X15.LE.1 

2.  X21+X24.LE.I 

3.  X31+X32+X33+X34.LE.1 

4.  X41+X42+X43+X44.LE.1 

5.  X51+X54.LE.1 

6.  X61+X42+X43+X64.LE.1 

7.  X71+X72+X73+X74.LE.1 
3.  X81+X82+X83+X84.LE.I 
9.  X91+X92+X93+X94.LE.1 

13.  X1B1+X132+X103+X134.LE.1 
+ 

»  COMPULSORY  PROGRAM  ELEMENT  WITH  LONER  BOUND 
+  DECISION  PACKAGE 

+ 

+  AT  LEAST  DECISION  PACKAGE  X33  HAS  TO  EE  CHOSEN 

+ 

11.  X33+X34-1 

+ 

i  ALTERNATE  COMPULSORY  PROGRAMS  WITH  LONER  BOUND 

<  DECISION  PACKAGES 

+ 

+  EITHER  AT  LEAST  DECISION  PACKAGE  X62  OR  AT  LEAST 
+  DECISION  PACKAGE  X42  HAS  TO  BE  CHOSEN 


12.  X42+X63+X64+X4Z+X43+X44.CE.1 


+ 


1 


A  -  - 
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i 


BUDGET  CONSTRAINTS 


♦  BUDCcT  YE'.R  1 

i 

13.  7X11+14X12+27X13+38X14+51115  +16I2!+3tX24 
+13X31+28X32+52X33+61X24  ^6X41+1 1X42+22X43+34X44 
+102X51+124X54  +11X61+21X62+42X63+51X64 
+9X71+19X72+37X73+48X74  +5X31+11X82+22X83+39X84 
+13X91+21X92+26X93+49X94  +64X131+79X102+88X133+131X10 
.IE. 353 

+ 

*  SUDCET  YEAR  2 

+ 

14.  3X11+15X12+28X13+39X14+42X15  +36X21+40X24 
+14X31+29X32+57X33+67X34  +7X41+12X42+24X43+26X44 
+138X51+129X54  +12X61+23X62+45X63+54X64 
+18X71+21X72+41X73+52X74  +7X81+12X82+22X83+39X84 
+18X91+18X92+24X93+42X94  +67X101+82X132+91X183+182X13 
.LE.343 

j 

+  SUDCET  YEAR  3 

+ 

15.  9X11+16X12+33X13+43X14+43X15  +42X21+43X24 
+16X31+31X32+63X33+74X34  +8X41+13X42+22X43+38X44 
+115X51+134X54  +13X61+24X62+49X63+57X64 
+11X71+23X72+43X73+56X74  +8X81+13X82+23X83+48X84 
+7X92+8X92+12X83+26X94  +71X131+85X102+93X183+133X134 
.LE.343 

+ 

+  BUDGET  YEAR  4 

+ 

16.  11X11+18X12+32X13+42X14+43X15  +48X21+49X24 
+17X31+35X32+69X33+81X34  +10X41+15X42+29X43+40X44 
+121X51+139X54  +13X61+23X62+40X63+49X64 
+14X71+24X72+46X73+68X74  +9X81+15X82+24X83+48X84 

+74X181+88X102+96X1S3+101X104 

.LE.336 

+ 

+  BUDGET  YEAR  5 

+ 

17.  12X11+19X12+34X13+43X14+43X15  +58X21+52X24 
+19X31+38X32+76X33+89X34  +11X41+17X42+32X43+41X44 
+129X51+142X54  +8X61+18X62+26X63+29X64 
+16X71+26X72+49X73+63X74  +12XS1+17X82+25X83+41XG4 

+78X101+89X102+99X133+100X134 

.LE .341 
PRINT 

LIMIT  533000 
OPTIMIZE 
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